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INTRODUCTION 



This document was originally prepared as a resource for those 
individuals involved in the planning and implementation of eval- 
uation activities related to N and D programs. Region I Techni- 
cal Assistance Center, in conjunction with the other Chaper 1 
Regional Centers, developed this handbook. 

Each office in the Department of Education has a subcommittee on 
Correctional Education. The purpose of these subcommittees is 
to work with program staff on concerns and issues of a specific 
nature to each office and to promote communication, support and 
delivery of educational services in corrections* One of the 
activities of the subcommittee is to prepare documents describ- 
ing the programs and services within each office. It was. felt 
that such a document would be a valuable resource for correc- 
tional education programs. 

Sincere thanks is extended to Dr. Lawrence Davenport, Assistant 
Secretary of Elementary and Secondary Education, for his support 
and promotion of his office staffs 1 involvement in Correctional 
Education. Mr. James Evans, Special Assistant, should be recog- 
nized for his leadership role as chair of the OESE subcommittee. 
Specific acknowledgment is also extended to Ms. Delores Hartman 
who works in this program area and who submitted this document 
for publication and dissemination. 

Additional inquiries may be addressed to: 

Ms. Delores Hartman 
Chapter I, Western Branch 
ROB-3, Room 5114 
7th and D Streets, S.W. 
Washington, D.C. 20202 

(202) 245-2214 

We hope that you find this document of value. For information 
on other available documents or on the Corrections Education 
Program in the U.S. Department of Education, please contact: 

Dr. Dianne Carter 

Office of the Assistant Secretary, OVAE 
U.S. Department of Education 
Reporters Building, Room 627 
400 Maryland Avenue, S.W. 
Washington, D.C. 20202 

(202) 732-2265 
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FOREWORD 



This handbook, The Eva 1 uat or 1 s Reference for Chapter ^1 
Neglected or Delinquent Youth Programs , has Teen developed 
as a resource for those individuals involved in the planning 
and Implementation of evaluation activities related to N or 
D programs. The evaluation process can often be a confusing 
one, resulting in information that 1s not of particular use 
or value to those Involved in the day-to-day program 
activities. This may be even more true for Chapter 1 N or D 
programs. It 1s hoped that this handbook will assist the 
user in planning and conducting evaluations which are 
feasible and which will provide useful information for 
program Improvement and reporting purposes. 

As most people are aware, there has been an intensive 
movement over the past eight years to improve the quality of 
evaluation data being reported to Federal and state educa- 
tion agencies, to Improve evaluation practices at the state 
and local level, and to Increase the utilization of evalua- 
tion Information 1n the Improvement of educational programs. 
ESEA Title I has been the vanguard of this movement through 
Its efforts to establish the Title I Evaluation and Report- 
ing System (TIERS), to encourage Title I programs to follow 
technically sound guidelines for Implementing evaluation 
models, and to establish regional Technical Assistance 
Centers to provide free consultative expertise in evaluation 
to state and local education agencies. While the majority 
of emphasis was placed on TIERS and evaluation models 
suitable for Title I programs in mathematics , reading and 
language arts in grades 2-12, the Department of Education 
did Initiate studies designed to explore whether comparable 
evaluation models could be developed for Title I Migrant 
Education, Early Childhood, Non-Instructional and Neglected 
or Delinquent Programs. 

When it became clear that, at least in the case of Title I 
Neglected or Delinquent Programs, it was not going to be 
practical or reasonable to develop strict program evaluation 
models and that in most cases the existing Title I models 
were not suitable, the Department of Education asked the 
Region I Tftle I Technical Assistance Center to lead an 
effort to develop an evaluation guide or reference for N or 
D evaluation practices. 

The Region I TAC called on assistance from other Title I 
Technical Assistance Centers across the country who had 
experience assisting N or D programs, had special expertise 
in testing or Instrumentation or had worked extensively in 
the area of evaluating program Implementation. 
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This handbook, T h e Evaluator's Reference for Chapter 1^ 
Neglected or Delinquent Youth l*ro"grams 1s a product of the 
efforts of staff from the: 



Region I - TAC 
Region II - TAC 
Region III - TAC 



(RMC Research and the University 
of Rhode Island) 

(Educational Testing Service 1n 
Princeton, NJ) 

(NTS Raleigh-Durham, NC) 



Region IV - TAC 



Region V - TAC 



Region VIII, IX, X - TAC 



(Educational Testing Service 1n 
Atlanta, GA) 

(Educational Testing Service 1n 
•Evanston , IL) 



(Northwest Reg1 ona 1 
Lab 1n Portland, OR) 



Educatl onal 



During a meeting 1n Washington, D.C. 1n the late spring of 
1982, attended by representatives from the U. S. Department 
of Education, the Title I TACs and state and local Title I N 
or D program evaluators and directors, a draft outline for 
the handbook was developed. From the discussions during 
this meeting several key points became apparent regarding 
the handbook and N or D evaluation: 



• No model(s) or reporting system would be devel- 
oped or suggested for N or D program evaluation 
at this time. 

• The unique characteristics of N or D programs 
and clients made the use of TIERS and existing 
Title I models Inappropriate 1n most cases. 

• Any support document such as this handbook 
should present sections that address issues 
related to program Implementation and Improve- - 
ment, as we 11 as the reporting of student 
outcomes and test scores. 

• There should be some logical flow to the 
handbook that would allow a person Inexperi- 
enced 1n evaluation an opportunity to address 
simple, basic Issues related to the evaluation 
of their N or D program and, with assistance 
from a TAC or other sources, conduct a reason- 
able program evaluation that would meet their 
needs and resources. 
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• Finally, the handbook would be only the first 
step 1n helping Chapter 1 N or D programs 
Improve their evaluation practices. In order 
for this handbook to have maximum effect 1t 
would have to be supported by assistance from 
other resources that would expand on the areas 
Introduced by the handbook. 



It 1s not expected that a user will read through this 
handbook from cover to cover at one time. Rather, the 
handbook has been organized so that the user can go 
directly to any topic of Interest (e.g., developing 
evaluation questions, Instrumentation), However, the 
user can also choose to start at the beginning and 
systematically develop an N or D program evaluation with 
little external assistance. 



This handbook 1s organized Into eight sections, each of 
which 1s briefly described as follows. 



1. PROGRAM EVALUATION: AN OVERVIEW 

This section discusses the many different purposes for 
conducting a program evaluation. The intended purpose 
shapps^the entire evaluation, so it must be clarified 
before undertaking any activities. A wide variety of 
evaluation purposes 1s discussed, ranging from accounta- 
bility to determining staff effectiveness. 



2. DESCRIBING THE PROGRAM 

This section has been Included to help the user develop 
a complete program description, which can then be used 
for fulfilling Information needs, for planning activi- 
ties, and for developing evaluation plans. Rather than 
present a model description, a difficult task because of 
the variety across N or D programs, 16 possible elements 
of a description are presented. These program elements, 
as will be seen, are then used 1n conjunction with the 
Identified evaluation purpose to plan the actual 
evaluation. 

li 
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DETERMINING THE FOCUS OF EVALUATION ACTIVITIES 



This section describes how the user can narrow the 
evaluation activities first by Identifying program 
elements which are of Importance and then by developing 
specific evaluation questions about the key elements. 
Because most programs will not have the financial 
resources and staff time to evaluate everything, 1t 1s 
necessary to determine where to place resources in order 
to obtain the most useful Information for program change 
and Improvement. 



STRATEGIES AND TECHNIQUES FOR PROGRAM EVALUATION 

This section presents a variety of strategies and 
techniques which can be used to collect evaluation 
Information. Rather than recommend one approach, it is 
left up to the users to decide which techniques are 
compatible with their N or D program characteristics, 
the resources available for the evaluation activities, 
and the user's preferences toward quantitative or 
qualitative approaches. Norm-, and cri teri on- reference d 
testing approaches are discussed, along with alternative 
data collection techniques, Including observations, 
questionnaires, interviews, and the use of existing 
records. 



INSTRUMENTATION 

This section provides Information that will aid the user 
1 n selecting the appropriate 1 n st rumentat 1 on • An 
annotated bibliography presents pertinent information 
and characteristics of various norm- and criterion* 
referenced tests, affective measures, item banks and 
other measures which might be appropriate for N or D 
evaluation activities. Because the bibliography had to 
be limited in length, a 11st of additional references to 
aid 1n the identification of instrumentation has also 
been 1 ncluded. 



RECORDKEEPING FOR PROGRAM EVALUATION AND MANAGEMENT 

This section deals with recordkeeping as a critical 
aspect of the overall evaluation and management of a 
program. The various types of records and how they can 
contribute to student management, to short- and long* 
range planning, and to evaluation and administrative 
reporting requirements are covered. 
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7. REQUIREMENTS AND RECOMMENDATIONS FOR EVALUATING CHAP- 
TER 1 N OR D PROGRAMS 



This section has primarily been included as a place for 
the user to Insert specific regulations and other 
frequently referenced Information. In this way, all of 
the documentation and Information of relevance will be 
in one easily accessible location. Additionally, an 
outline of recommended information which might be 
collected by each program for o vera 11 summary purposes 
is included. 



8. RESOURCES FOR N OR D PROGRAMS 

This section presents some additional resources which 
a re available to project personnel. The Technical 
Assistance Center services, including on-site visits, 
local workshops, telephone consultations, packaged 
materials, and the Clearinghouse a re discussed. The 
user is also briefly Introduced to tne National Dif- 
fusion Network . 

Most of the previously described sections in the handbook 
also Include a variety of appendices. These appendices are 
typically forms, checklists, steps to be followed or more 
detailed descriptions of a topic which was introduced in the 
section. For example, the appendices after Section 5, 
Instrumentation, Include a criterion-referenced test rating 
scale. Another appendix 1n this same section summarizes 
guidelines which help determine when to test out-of- le ve 1 . 
These appendices should be duplicated and used as needed. 

We hope that this handbook will be a valuable resource for 
Individuals Involved in the evaluation of Chapter 1 N or D 
programs. As new materials become available or existing 
Items change, we will be distributing the upgraded contents. 
In the meantime, should you have questions or comments about 
this handbook, please contact your regional Technical 
Assistance Center. 



Everett Barnes, Jr. 
D1 rector 

Region I , Chapter 1 
Technical Assistance Center 
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1. PROGRAM EVALUATION: AN OVERVIEW 



PURPOSES OF EVALUATION 



Before undertaking the evaluation of a program, one should 
have a clearly defined purpose in mind. Different purposes 
suggest different evaluation questions, designs and tech* 
niques. The purpose selected really shapes the evaluation 
and greatly influences the types and uses of evaluation 
results. In general terms, evaluation can serve to comply 
with requirements, to find out more about how a program is 
operating, or to identify effective practices and to improve 
less effective ones. 



Within those broad general categories of intent, there are 
more specific purposes which might serve as focuses for the 
evaluation of programs for Neglected or Delinquent youth. 
Examples of those purposes are: 



Accountability and reporting. 

Determining how best to match services with 
individual student needs. 



• Determining the degree to which a program has 
been implemented. 

• Assessing short-term effects of programs. 

• Assessing long-term effects of programs. 

t Identificationand description of effective 
practi ces . 

• Identification of relationships cmong services 
and prog ram components. 

• Examining management and staff effectiveness. 



It should be noted that purposes will often overlap and that 
one evaluation may result in information for more than one 
purpose . 
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While not exhaustive, the previous list may help to define 
the purpose or purposes of an evaluation for a particular 
program. Each purpose is briefly described as follows, 
along with some reasons why it might be selected as a focus 
for evaluation activities. 

It is probably not realistic to evaluate all aspects of a 
program at the same time. By focusing attention on a spe- 
cific purpose and on one or two aspects or components of a 
program, an evaluation is mo re likely to yield useful 
information. Once an evaluation purpose has been defined 
and aspects or components of interest have been selected, 
the next step is the formulation of specific questions which 
the results of the evaluation should answer. The develop- 
ment of evaluation questions, along with sample questions, 
is discussed in Section 3. 



Accountability and Re porting 

It is often necessary to evaluate some aspects of a program 
to ensure that the program is in compliance with the re- 
quirements or expectations of a governing board or funding 
agency. Usually, the requirements of agencies are based on 
law and regulations, and frequently the results of such an 
evaluation must be reported in a specific form on a regular 
basi s • 

If accountability and reporting are the primary purposes for 
evaluating a program, it will probably be necessary to focus 
the evaluation on determining whether the information needs 
of the boards or agencies which receive results are being 
met. Specific reporting requirements will dictate minimum 
evaluation activities. This type of evaluation usually 
relies heavily on accurate and complete records* See Sec- 
tion 6 for information on recordkeeping. 



Determining How Best to Match Services with Individual Stu- 
dent Needs 

Most programs are designed to meet some specific range of 
student needs. Tailoring program activities to such student 
needs is a complex, ongoing task. As student populations and 
needs change, program activities must also Qhange. 

One purpose of an evaluation might be to examine or re- 
examine the range of needs being addressed by the program 
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1n light of the needs present in various segments of the 
student population. Also important for study would be ways 
in which student needs are identified and how well existing 
program activities and materials are meeting those needs. 
This type of evaluation activity might be undertaken if 
there is concern about instructional effectiveness, the 
level of student satisfaction, or long-term effectiveness of 
the program. 



Determining the Degree to Which a Program Has Been 
Implemented 

It is not uncommon to find that a program is operating quite 
differently from the way it was intended to operate. One 
purpose of an evaluation, therefore, might be to examine how 
many key elements of the program are actually in place. If 
an evaluation of the short- or long-term effects of a 
program is planned, the program's level of implementation 
should also be evaluated. The results of an evaluation of 
effects will only be meaningful if there is some assurance 
that the intended program is in place. An evaluation of 
program implementation is especially important when new 
programs are undertaken or new staff are added to 'the 
program. 



Assessing Short-Term Effects of Programs 

Typically an evaluation of short-term effects focuses on 
changes in students during the course of the program. Such 
changes may be in cognitive or in other skill areas. 
Change is generally compared to pre-program behavior or to 
the behavior of students who are not in the program. This 
type of evaluation activity can help to point to general 
prog ram strengths and weaknesses and is often used for 
accountability purposes. 



Assessing Long-Term Effects of Programs 

Someti mes the effects of a program are most clearly seen 
after a student leaves the program. Often much can be 
learned about a program by examining the experiences of 
former participants. Programs can also benefit from viewing 
their effectiveness across several years' operation. When 
post-program behavior or several years of a program are 
being studied, the focus of the evaluation is said to be on 
long-term effects. This type of evaluation activity might be 
considered when the effects of the program are intended to 
appear in later job or training performance. 
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Identification and Description of Effective Practices 



Some methods are bound to be more effective than others. An 
evaluation of short-term or long-term program effects may 
point to an area of effectiveness which requires further 
investigation. That investigation could include a review of 
key program elements and a determination of which elements 
contribute most to overall program effects. 

Evaluating th« impact of different techniques and practices 
on various typss of students can produce i nf ormat i on .whi ch 
will greatly enhance the effectiveness of a program. It is 
also important that the dissemination of successful methods 
be considered, along with jse of information from others 1 
expe ri ences . 



Id entification of Relationships Among Services and Program 
Components 

Often programs are designed and implemented without adequate 
consideration of how they might be integrated with existing 
programs. One worthwhile focus for an evaluation might be to 
examine the interrelationships among programs or program 
components in order to identify areas of overlap, to redis- 
tribute resources, to prevent trapping students in competing 
or conflicting situations, or to best match programs with 
•students' needs. This type of evaluation activity should be 
considered if there 1s confusion about the functions of some 
components, if key elements are the same across several 
programs, or if there is difficulty 1n matching students' 
needs to programs. 



Examining Management end Staff Effectiveness 

The effectiveness of a program can sometimes be improved by 
making better use of staff, enhancing individuals' skills, 
improving staff communication and morale, or altering 
management practices. Focusing evaluation activities on this 
area suggests the assessment of staff strengths and needs 
and also the assessment of the short-term effects of 
specific management practices that have been instituted. 
This type of evaluation activity should be considered when 
there is friction among staff either in or between programs, 
when there is difficulty in the implementation of programs, - 
or when personnel changes are frequent. 
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SITUATIONAL CONSTRAINTS 



There a re many situations peculiar to an N or D setting 
which 1 1 m 1 1 the procedures that can be used to evaluate 
educational programs* It 1s Important to be aware of such 
situational constraints when planning the evaluation. 
Although adaptations may be required to help reduce the 
effect of any constraints, good evaluation procedures can 
still be practiced In N or D settings. Furthermore, taking 
existing constraints Into consideration before establishing 
the evaluation procedures will help ensure that the evalua- 
tion results are meaningful. 

In general, constraints include those situations, regu 1 a- 
tions or characteristics which nothing can be done about; 
plans have to be made around constraints. There are, for 
examp le , some characteristics of N or D institutions that 
severely hamper the Implementation and evaluation of 
programs within that setting. There are also problems that 
can reduce the effectiveness of instruction in the .N or D 
setting. Finally, there is always a variety of miscellan- 
eous constraints which can directly impede any evaluation 
process. But as long as the existence of certain con- 
straints is known, the evaluation can be planned to reduce 
their effect. The remainder of this section deals with some 
constraints which are common to many N or D programs. 



Transient Student Populations 

Perhaps the most severe limitation to evaluation is the 
transient nature of the student population. Turnover in N 
or D settings is often high, resulting in a short duration 
of instruction. For a variety of reasons, a student's stay 
1n the program may be reduced to a minimum. Before students 
enter the actual program, they often go through a reception 
center for observation and testing. Then, once assigned to 
a program, students may go through an orientation program to 
familiarize themselves to their new roles and situation. 
This process further reduces the time a student will sp?nd 
in the actual program. While the average length of si ay 
Mes among settings and from state to state, a national 
study of N or D programs found that the average student 
received four months of Instruction. Some states, however, 
have average lengths of stay as short as 2.9 months. 

The problem of turnover is compounded by students entering 
or leaving the program on the basis of institutional needs 
rather than on educational needs or progress. Often, the 
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Instructor has little or no notice that a student is about 
to enter or leave the program. These types of constraints 
clearly affect evaluation activities. For example, it 
becomes more difficult in such situations to implement 
evaluation activities which require pretest and posttest 
scores for all students. 



Institutional Requi rements 

Education often has a low priority within N or 0 programs. 
Because of this low priority, students often miss class. The 
students may be needed for work details or may be locked up 
for disciplinary reasons. In addition, security measures 
within the institution can interfere with both instruction 
and evaluation. Evaluation activities must be planned to 
take into account any specific institutional restrictions. 



Class Time 

Compounding the problem of a short time spent in the 
program, two other problems further reduce instructional 
time: absenteeism and less than full use of class time. 
Absenteeism may occur for many reasons, such as student 
illness, disciplina ry measures, institutional requi rements, 
or rehabilitation (such as visits to the rehabilitation 
worker). Less th?n full use of class time may also occur if 
the instructor is absent and class is cancelled because a 
substitute is not available or a substitute may be 
present, but without adequate lesson plans, resulting in 
misused time. 

Even with the students and instructors present, class time 
is often consumed by non -1 n st ruct 1 ona 1 activities. For 
example, one national study found that 37 percent of class 
time in N or D institutions was spent on non-instructional 
activities. The evaluation must be planned to take these 
constraints into consideration. The use of class time may 
even become the focus of the evaluation (see the discussion 
of time-on-task in Section 4). 



Achievement Level s 

Another set of problems can result when the stu dents a re 
functioning far below their age expectancy, N or D students 
often have a history of failure, thus student motivation 
tends to be low. Since there is a lack of high interest-low 
ability materials to teach basic education skills, the 
students often use materials- developed for younger students. 
This lack of appropriate materials can, in turn, have a 
negative effect on student progress, 
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Testing Conditions 



Testing conditions may be less than ideal, especially if 
tests are given during the first few weeks the student is in 
the institution. Testing may take place in a diagnostic 
center where the student has already been given a series of 
other tests. Also, the student is probably not in the best 
frame of mind at this time a factor which may further 
contribute to an inaccurate measure of ability. In addi- 
tion, it has been noted tha', some students, familiar with 
the system, will suppress their test scores so that they 
will be placed in special programs or so that they can more 
easily show educational progress. 



Appropriate Measurement Instruments 

Most of the tests being used for evaluation of programs in N 
or D settings were designed for use with average students in 
non-institutional settings. Test norms often /do not extend 
to adolescents who are functioning at lower educational 
levels. There are, however, alternatives to using commonly 
available standardized tests. There are also some instru- 
ments available, which measure progress in adult basic 
education curricula. (For more information on assessment 
instruments, see Section 5.) 

Another constraint in regard to testing is that standardized 
tests are frequently developed to measure progress from fall 
to spring. Many tests sample broad bands of achievement and 
are not sensitive enough to measure progress in the brief 
periods of time N or D students participate in some pro- 
grams. Furthermore, the test norms have been developed for 
specific times <?uring the year; for test scores to be 
meaningful, the N or D student must usually take the test 
during the same period. Any variations can affect the 
evaluation results. 



Procedures for Tracking Students 

Students 1 previous school records can be difficult to 
obtain, if they are accessible at all. Often the students 
have been out of school for quite a while and so have no 
records, the time it takes to track down any records is 
also a problem for the many students who stay in the program 
for a short time. Post-release information necessary to 
evaluate the long-term effectiveness of programs 1s equally 
hard to obtain. Parole officers are often the best equipped 
to gather follow-up information about students who have left 
the program, but large caseloads may prevent them from doing 
so. 
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Appropriate Models for Evaluation 



Clearly an overriding problem is the lack of evaluation 
models that take Into account the various constraints 
discussed. For that reason, the evaluation of N or D 
programs requires more flexibility in selecting and imple- 
menting evaluation techniques. The remaining sections of 
this handbook are designed to suggest some possible evalua- 
tion alternatives. Since there are no easily applicable 
evaluation models available, procedures which can be used 
for the evaluation of programs in spite of the possible 
constraints in N or 0 settings will be discussed. 
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2. DESCRIBING THE PROGRAM 
THE RELEVANCE OF A COMPLETE PROGRAM DESCRIPTION 



Developing a complete and accurate description of an opera- 
ting program 1s not a simple task. It requires an unbiased 
view, careful attention to detail and an understanding of 
the role each part of the program plays within the whole 
system. It 1s not surprising then that few educational 
programs have routinely developed program descriptions which 
extend beyond an abstract or a list of program elements. 

A complete program description 1s useful in several ways. 
First, 1t 1s a vehicle for clear communication a bout the 
program both externally and Internally. A description serves 
the Information needs of funding and governing agencies or 
boards; accountability begins with a common understanding of 
the Intentions of program designers. In a similar way, a 
complete program description can serve Internal staff 
communication needs as well. A written guide to the program 
answers questions of new and old staff 1n a definite manner 
that can be supplemented with, but not replaced by, the 
collective oral history of staff members. Potential prob- 
lems are prevented by the existence of clear procedures. 
Decision-making about new Issues 1s facilitated when all 
parties can make reference to common information. The 
potential for developing creative solutions to problems is 
Increased when staff members can spend less time and energy 
reinventing policies and procedures as they are implementing 
the program. 

Second, complete program descriptions are invaluable in the 
planning process. Certainly, if a new program is being 
undertaken, clear descriptions of all major elements will 
facilitate Implementation. Intel li gent planning for re- 
source a 1 location with expanding or contracting budgets 
demands accurate descriptions of program Intentions and an 
understanding of how all elements function. Coordination 
across different programs serving Neglected or Delinquent 
students 1s one of the more difficult tasks faced by program 
managers. Locating and eliminating areas in which programs 
overlap and Identifying gaps 1n services re qui res de tailed 
descriptions of program services and functions. 

Finally, a complete program description is the basis for the 
development of evaluation plans which yield useful results. 
The development of evaluation questions (see Section 3) is 
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grounded in a knowledge of the intentions and operation of 
program elements. Further! a thorough description of 
program elements is necessary to frame realistic recommenda- 
tions for program modification based on the results of 
eva 1 uat 1 ons . 



THE SIXTEEN PROGRAM ELEMENTS 



It 1s clear then that a good program description will "serve 
a variety of important functions. Because N or D programs 
vary greatly, 1t is not possible to develop a model descrip- 
tion that would be applicable to most programs. Instead, 
this section includes guidelines for developing descriptions 
of sixteen major program elements which would be 'common to 
many projects* Each element is discussed and the components 
of a description are listed. 

It is unlikely that any one program description would' 
include a lengthy narrative about each element. The 
previous discussion of purposes . suggests that one might 
develop a description of an element if any of these condi- 
tions were true: 



t An evaluation focusing on certain program areas 
is being planned- (e.g. , a study of time spent 
on instructional areas). 

t There is concern or confusion about an element 
(e.g., no one is sure about student selection 
procedures ) . 

t Changes in an element have been suggested or 
are planned (e.g., a new testing procedure is 
to be inaugurated). 

t Major staff or administration changes are about 
to occur. 

t Outside support is being sought for a program. 



Another option is to build a complete program description 
over time, selecting a few elements to describe thoroughly 
during each program year. This can be done by first using 
the Program Element Checklist (Appendix 2-A) to review 
current program descriptions for completeness. On the basis 
of the results of the checklist, additional program element 
descriptions can be developed where needed. 
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!• Admi n1 strati on 



Administrative activities include those policies, procedures 
and routines that are required to operate programs and 
services effectively. It is important to specify adminis- 
trative activities so that expectations are clearly known to 
all staff and coordination among program elements is 
possible. Clarity of expecta t i ois and coordination ensure 
smooth program operations. A description of administration 
includes: 

t procedures for supervision and evaluation of 
staff 

• procedures for ensuring effective communication 
of information to staff and to others (funding 
agency, other services, etc.) 

• hierarchy of reporting relationships 

• budget authorization policies 

• staff recruitment and hiring practices 

• policies for grievance and conflict resolution 
related to students and staff 

• long-range and short-range planning procedures 

• establishing a documentation system that sup- 
ports compliance with regulations 

• procedures which ensure the health and safety 
of staff and students 

• procedures for obtaining and upgrading staff 
benef 1 ts 

• administrative roles and responsibilities 



2. Staffing 

The element of staffing encompasses four areas: (1) 
staffing patterns, which includes job categories and 
student-staff ratios; (2) responsibilities for each job 
category; (3) staff background and qualifications; and (4) 
staff development. The delivery of any program depends in 
large part on the quality of staff; quality can be enhanced 
by providing sufficient numbers of staff, selecting those 
with appropriate experience and ba c k g round s i p rov i d i n g 
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adequate support services and carefully matching backgrounds 
with tasks required. A description of staffing patterns 
i nc 1 udes : 



t types of job categories (e.g., instructor, 
aides, counselor,- coordinator) 

t numbers of full and part-time staff employed in 
each category 

t overall student-staff ratios 

t specific program student-staff ratios (e.g., 
vocational training, basic skills) 

A description of responsibilities for job categories 
includes minimum expectations for performance related to: 



t instructional, administrative, and non- 
ins tructi ona 1 duties 

• recordkeeping, evaluation and reporting 

t maintenance and upgrading of professional 
skills 

• student management tasks 

t special committee assignments (e.g., providing 
input for hiring and review of other staff) 

A description of staff background and qualifications 
includes : 

t a summary of experiential and academic prep- 
aration of staff members by job category 

t unique requirements of job responsibilities and 
how staff characteristics match those require- 
ments 

t a summary of special interests and abilities 
A description of staff development includes: 
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the process by which staff training needs are 
identified 

in-service activities planned to meet those 
needs 

evaluation procedures for in-service activities 

examples of training needs which have been 
identified 

examples of the types of in-service activities 
which are conducted 



3. Budget 

A budget description includes the dollar amounts to be 
expended for specific program purposes. Detailed budget 
descriptions provide both the proof that resources have been 
allocated to match major goals of the program and the 
guidance required to ma'.e choices among competing expendi- 
ture alternatives. Include in a budget description: 



• the costs projected by line-item category 
(e.g., equipment, salaries for instruction, 
instructional supplies) 

• the costs projected by general purpose (e.g., 
academic activities, vocational training, 
counseling program) 

• a projection of costs per student 

• a description of costs which are one-time 
expenditures (e.g., program start-up costs, 
facilities) 

• authorization to spend policies 

t a description of budget planning procedures 

• internal fiscal accounting p rocedu res 

• fiscal reporting schedule 
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4 . Student Referral Proce s s 



Student referral 1s the process by which students who are 
potentially eligible for a program are identified. A 
well-defined referral process ensures that those students 
most in need and whose needs are most in accord with the 
program become the eligible candidates for the next steps in 
the process. A well-defined process produces better 
referrals than a casual system. It also ensures that 
students who may not seem appropriate for the program Under 
consideration are referred to another service or program. A 
description of the student referral process Includes: 



• deslynatlon of who refers students (staff 
members or outside agencies such as other 
Institutions, the courts) 

• the means by which referring individuals 
receive sufficient information about the 
program upon which to ba«e a referral (e.g., 
written or oral program descriptions, memos, 
Individuals who are responsible for generating 
referrals from others) 

• approximate time schedule for receipt of refer- 
rals 

• content of the referral, Including whatever 
judgments or Information must be provided by 
referring Individuals 

• any forms used in the process 



5. Student Selection Criteria 

Student selection criteria are the formal standards by which 
students are accepted into a program or service. It is 
Important to remember that criteria do change as programs 
and services develop; criteria should reflect needs assess- 
ment findings and the programs designed to meet those needs. 
A clear description of student selection criteria ensures 
compliance with regulations and allows a determination of 
whether standards are fair and appropriate. Well-defined 
criteria ensure that students 1n the program are those whose 
needs are most in accord with the program. A complete 
description of the student selection criteria includes: 
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• a description of the relevant characteristics 
of the target audience as determined by needs 
assessments (e.g., achievement and demographic 
i nf ormatl on ) 

• the process by which a pool of eligible stu- 
dents 1s Identified 

• the procedure by which the most educationally 
needy within that pool are rank-ordered 

• the selection indicator(s) used in the above 
procedure (e.g., tests, instructor referral', 
self-referral ) 

• means of obtaining Information for indicator(s) 

• the way in which Indicators are combined to 
Identify each student's degree of need (com- 
posite, multiple cut-offs, single criteria) 

• the actual cut-off scores used (e.g., 25th 
percentile, 65 out of 100 points on a composite 
score, or three of five indicators of need show 
eligibility) 

• waiting 11st policy for filling open slots 

• exit criteria which specify expectations of 
success or mastery 



6, Institutional Goals 

Institutional goals are statements which describe the 
desire^ outcomes obtainable by the program as a whole. These 
goals may encompass a wide varievy of areas, such as: 
upgrading or adding new facilities; restructuring the 
program to meet projected changes in stuoent needs; estab- 
lishing new funding sources; or strengthening positive 
community attitudes toward the program. Institutional goals 
are typically more long-term in nature and may require the 
involvement of a variety of program personnel. A description 
o* institutional goals should Include statements specifying: 



• the goal and how reaching it will enhance the 
program 

• a timeline for attaining each goal, including 
the various tasks to be performed 
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• who 1s responsible for coordinating the activi- 
ties required to reach each goal 

• an Identification of other personnel who will 
be Involved 1n the activities 

• a way to evaluate whether the goal has actually 
been reached 



7 • Student and Staff Goals 

Student and staff goals are broad statements of the outcomes 
obtainable by students and staff through planned activities. 
Each goal may Include a variety of cognitive or skill 
outcomes and may be either short-term or long-term in 
nature. Typically, one goal will encompass a group of 
measurable objectives. All or only some of the goals may be 
attained depending upon Interests, abilities and time 
available. A description of goals should include statements 
sped fy1 ng : 



• goals categorized by r.arget audience 

• who is responsible for developing new goals and 
revising existing ones 

• where written copies of all goals are filed 

• how goals are used to plan Instructional objec- 
tives, activities, and purchase of commercial 
ma terl a 1 s 

• how goals are used to develop individual stu- 
dent plans 

• who receives copies of goals and how this 
information 1s used by recipients (I.e., 
potential employers , counselors) 

• how goals are used in Initial test selection 
activities 

• how goals are used to monitor overall student 
progress 

• how goals are used to monitor staff development 

• how goals are used 1n evaluation activities 
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8. Student and Staff Objectives 



Student and staff objectives are measurable statements of 
the outcomes obtainable through planned activities. As with 
goals, ob jec t1 ves . may be cognitive or 1n other skill areas. 
The mastery of one or more objectives should lead to 
attainment of specific goals. Objectives may cover a fairly 
broad range of outcomes or may focus on a single, rather 
narrow outcome. 

Objectives should always be stated in measurable terms and 
Include the following three parts: (1) the conditions a 
statement which describes the circumstances under which the 
outcome will be demonstrated; (2) the performance a 
statement which Includes one or more measurable verbs which 
describe the outcome; and, (3) the criterion a statement 
which specifies the minimum acceptable standard which must 
be reached 1n order to demonstrate mastery of the objective. 
A description of objectives should include statements 
specifying: 

• objectives categorized by target audience and 
goals 

• who 1s responsible for developing new objec- 
tives and revising existing ones 

• where written copies of all objectives a re 
filed 



o how objectives are used to develop, revise, or 
adapt activities 

• how objectives are used to organize the content 
and sequence of a program or course 

• how objectives are used to develop individual 
student plans 

• who receives copies of objectives and how this 
Information 1s used by recipients (i.e., 
potential employers, counselors, follow-up 
educational program personnel) 

• how objectives are used to develop or select 
tests 



• how objectives are used to monitor student 
progress 

• how objectives are used to coordinate activi- 
ties among courses within the institution 
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how objectives are used to coordinate activi- 
ties with follow-up and work placement outside 
of the institution 



t how objectives are used in evaluation activ- 
ities 

9. Developing Individual Student Plans 

Developing individual student plans involves the identifi- 
catlon of goals and objectives most suitable to meet each 
student's needs and then selecting or developing activities 
which will ultimately help the student reach the desired 
outcomes* A variety of Information can contribute to the 
development of a plan f Including diagnostic tests, achieve- 
ment tests, interest surveys, affective surveys, counseling 
sessions, background 1n f or mat 1 on, learning style preference 
surveys, and discussions with the student. Descriptions of 
Individual student plans should specify: 

t standard format for each plan 

t where plans are stored and how confidentiality 
is ensured 

t who 1s responsible for obtaining Information to 
be used 1n the plan 

• how and where various types of Information are 
obtained 

t when plans will be developed and updated 

• how plans are revised 1n response to un- 
scheduled occur re nces (such as a student not 
progressing or the availability of new in- 
formation ) 

t how plans are revised in response to planned 
growth and development (such as goals being 
met ) 

t who 1s Involved 1n the development of a plan 
(e.g., past instructors, present Instructors, 
counselors, parents, guardians, the student) 

10. Instructional Activities 

Instructional activities Include all of the planned learning 
experiences, media, and hardware used to teach objectives. 
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Within any setting, there should be a variety of deliberate- 
ly planned activities group discussion, simulations, 
print materials, fllmstrips and more each of which has as 
Its purpose to teach the student a specific topic, skill, or 
attitude. Instructional activities may include those 
developed by an Instructor, commercial materials or a 
combination of Items. 

Ideally, Instructional activities a re organized around 
sped f 1 c object 1 ves. Furthermore, in order to better meet 
Individual needs, there should be more than one approach 
available to teach any one objective. For example, students 
who have difficulty reading would benefit from instruction 
that 1s not dependent upon reading skills; audiotapes or 
high Interest-low readability texts would be more effective 
than standard textbooks. Or, since some students may 
require more repetition and practice than others, instruc- 
tional activities with additional practice exercises would 
be of benefit. 

Descriptions of Instructional activities should be developed 
for all aspects of a program, including academic, voca- 
tional, and real world survival skills. These descriptions 
should specify: 



• activities cross-referenr^d to goals and 
object 1 ves 

• commercial and staff-developed programs in use 

• how staff and students access materials 

• preferred instructional approaches 

• examples of activities 

• how staff work together to develop, adapt, and 
revise activities 

• orientation of new staff to instructional 
approaches 

• predominant types of equipment available 

• specific facilities employed 



11 • Coordination Aatong Courses in the Instructional Program 

Coordination among courses 1n the instructional program is 
necessary in order to provide a more cohesive and integrated 
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sequence of instruction. Just as 1t is necessary to care- 
fully and logically order goals and objectives within a 
course, the same care must be taken across courses offered 
within a program. Ensuring that this coordination is done 
a 1 lows instructors to eliminate unnecessary overlap across 
courses and at the .same time to develop course sequences by 
which students can move successfully from an entry level to 
a more advanced course. While a brief review may be appro- 
priate, a complete instructional sequence over previously 
learned topics would be a non-effective use of both instruc- 
tor and student time. On the other hand, if an advanced 
course assumes that certain entry level skills have already 
been taught, failure to teach those skills would mean, that 
the instructor of the more advanced course would have to 
spend time providing instruction that should have been 
taught elsewhere. 

Coordination across courses can also be used to Increase 
student motivation and progress. For example, students who 
are very Interested in their auto mechanics course may 
perform better in a basic math course if the math skills are 
taught in the context of mechanics. Similarly, reading 
skills might develop faster and with less resistance if 
taught through a driver's license manual or a repair manual. 
However, th.is coordination will not always occur unless it 
is deliberately planned; thus the plans for coordination 
encourage communication and ultimately result 1n a more 
successful experience for the students involved, 

A description of the process for coordination among courses 
should specify: 



• the plan for ensuring this coordination 

• who is responsible for organizing, monitoring, 
and conducting coordination activities 

t how individuals are identified for participa- 
tion 1n coordination activities 

t when coordination activities occur 

t the process for updating and revising linkages 



12 ♦ Coordination with Follow-up Education and Work Place - 
ment 

Coordination with follow-up education and work placement is 
just as critical as. coordination within the program. 
Although it is realized that coordination with follow-up 
placements is often not possible, it 1s desirable and so 1s 
discussed in this section. Because the ultimate purpose 
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of any program is student success at the next step, each 
instructional plan should be developed with the next 
placement in mind. The student will be more capable of 
success at this next step if the instructor is aware of the 
necessary entry-level capabilities. For example, if the 
student will be placed in a public high school work-study 
program, mastery of certain cognitive and affective skills 
will be critical for success. If a student 1s entering a 
vocational training program, a somewhat different set of 
skills may be necessary. A student who will be going 
directly into a job placement situation will have very 
different needs. 

The individuals responsible for administering the follow-up 
education or work program will be able to identify specific 
cognitive or other skill areas which are entry- level 
capabilities for their particular program. For example, a 
high school mathematics instructor will be best able to 
identify the entry-level mathematics skills. The job 
placement counselor will be more sensitive to behavior such 
as correct dress and appropriate interview skills. The 
parole officer will be able to identify specific skills 
which will help the student adjust to new situations. 
Coordinating and communicating with these types of people 
will not only help ensure student success but will also make 
them more willing to accept the students into their particu- 
lar program; they will have a better understanding of the 
students' capabilities. 

A description of the process for coordination with follow-up 
education and work placement should specify: 



• the plan for ensuring this coordination 

• who is responsible within the institution for 
organizing, monitoring, and conduct ing coordin- 
ation activities 

• how individuals within the institution are 
identified for participation in coordination 
activities 

• the process for identifying the follow-up 
education and work placement programs that 
should be linked to internal programs 

• the process for identifying and contacting 
other outside support systems 

• when coordination activities occur 

• the process for updating and revising linkages 
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13 . Student Management Techniques 



Student management techniques encompass the wide range of 
methods used to encourage growth on the part of the student 
1n cognitive and other skill areas. In addition to 
providing the student with activities which teach specific 
objectives, Instructors typically employ management tech- 
niques which motivate the student to learn. These tech- 
niques range from non-verbal rei nf orcenents such as a smile 
to very formalized token rewards for prespecified behaviors. 
Formal contracts, grades, and modeling are other examples of 
management techniques. 

Ideally, the management techniques used should be matched to 
the student; different individuals are motivated in differ- 
ent ways. For example, a younger student may be encouraged 
by a positive non-verbal gesture or positive verbal approval 
from the instructor. In contrast, an older student with a 
history of academic failure and poor interpersonal skills 
may do better if a more concrete system of rewards, such as 
accumulating points to buy free time, 1s implemented. A 
variety of management techniques may need to be tried before 
the most effective ones are Identified. If possible, the 
management techniques used for each student should be 
consistent from instructor to instructor. Descriptions of 
management techniques used should specify: 



• how managemeat techniques are designed for 
students who are involved 

• examples of techniques presently being used 

• institution-wide standard practices 

• schedule of periodic review for effectiveness 
of techniques 



14. Non-I nstruct i onal Servi ces 

Non -1 ns t ruct 1 ona 1 services include those program components 
primarily designed to promote health, safety, and other 
non-cognitive goals for students. Activities may or may not 
be directly related to Instruction. Types of services which 
might be included are individual and group counseling, 
follow-up work and education placements, preventive medical 
and dental care, recreational and leisure time activities, 
career placement, and family service activities. 
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It is especially important to develop complete descriptions 
of non-instructional services because their functions are 
not likely to be well understood by students, institutional 
staff not directly involved in the service, and those ex- 
ternal to the institution, including funding agencies. 
Descriptions will help to ensure that duplication of 
services does not occur, that student referrals are appro- 
priate, and that all students can take advantage of opportu- 
nities a va i 1 ab 1 e through i n s t i t u t i on s • A description of 
each non-instructional service includes: 



• goals and objectives or statements of purpose 

• examples of activities and materials 

• numbers and types of staff 

• numbers and characteristics of students in- 
volved 

• recordkeeping procedures 

• evaluation activities 



15, Recordkeeping 

Recordkeeping procedures are a critical aspect of the 
overall management of a program, especially in situations in 
which students may enter or leave a program at variable 
times during the year and in which students may come into 
the program with a wide range of backgrounds, capabilities 
and interests. Comprehensive records can actually be the 
key to a variety of activities, including: the coordination 
of individual and group progress through both the instruc- 
tional and non-instructional activities in the program; the 
conducting of short- and long-range planning activities to 
develop a program most responsive to student needs; the 
conducting of evaluation activities in order to determine 
program effectiveness; and the meeting of administrative 
reporting requ i rements • 

Comprehensive records should include the following types of 
info rm at ion: individual student files; individual student 
progress records; group progress records; a list of goals, 
objectives and activities cross-referenced by target audi- 
ence; an inventory of consumable and n on -con s uma b 1 e re- 
sources; a list of outside resources and types of services 
offered; staff files; fol low-up academic, vocational, and 
job placement opportunity; student attendance records; 
student selection procedures and documentation; results 
of past surveys administered to staff or students; follow- 
up data collected on students in academic, vocational and 
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job placements; interview documentation with potential em- 
ployers; results of past evaluation activities; state 
reports; and all other information which may form a basis 
for future planning or evaluation activities. 

A description of recordkeeping procedures should include: 



• an Identification of the types of records 
maintained and the individual (s) who are 
responsible for collecting and updating the 
nece ssa ry i nf or ma t i on 

• 

• an explanation of how the various records a re 
used by students, instructors, administrators 
and other program or outside personnel 

• samples of recordkeeping forms, where appro- 
priate 

16 . Program Evaluation 

Program evaluation is the process of systematically gather- 
ing information to determine the value or effectiveness of 
program elements and services. Evaluation can serve a 
variety of purposes. Section 1 of this document provides 
examples of evaluation purposes (e.g., determining short- 
term and long-term effects of projects, establishing the 
degree to which projects have been implemented). Further, 
evaluation activities can be of many different types (see 
Section 4 for descriptions of evaluation techniques espe- 
cially for N or 0 projects). Evaluations are typically 
designed to answer a set of questions which are considered 
important to staff questions which affect long- and 
short-term program development and operations. The evalu- 
ation results should form the basis for improving programs 
and services. All evaluation activities should always be 
described before any evaluation activity actually begins. 
The worth of evaluation is in Its use. Maximum use of 
information requires the systematic and careful collection 
and examination of information as well as the involvement of 
those who will be affected by the results. A good descrip- 
tion of evaluation plans is essential so that the results 
will be comprehensive, comprehensible, credible and useful. 
A complete description of the evaluation of each program 
includes: 



• a statement of the purpose of evaluation 

• key evaluation questions 

• management of the evaluation 

• evaluation design 

3.9 

2-16 



Appendix 2-A 
PROGRAM ELEMENT CHECKLIST 



The first step 1n developing a complete program description 
is to review, for completeness, the 1 nf ormat 1 on cu rrent ly 
available in written form about each major program element. 
The Program Element Checklist was designed to facilitate 
this review process* The checklist is organized by the six- 
teen major program elements which were discussed in Section 
2. Each major element 1s further subdivided Into specific 
topics. 

• To use the checklist, rate the current status 
of documentation about the topic under consid- 
eration. If a description exists and is both 
adequate and accurate, place a check mark in 
the column labeled YES. 

• If the topic 1s either not described in written 
form, or 1f existing descriptions are inade- 
quate or Inaccurate, place a check mark in the 
column labeled NO. 

• For any topics where a NO has been checked, use 
the column labeled NOTES to Indicate what needs 
to be done in the way of additional documenta- 
tion or modification 1n order for the descrip- 
tion of the topic to be complete. Use the 
NOTES column to also Indicate what partial 
Information exists, 1f any, where further 
Information might be located, or who might be 
Involved In developing the description. 

t If the topic 1s not a part of the program or 1s 
not applicable 1n some way, place a check mark 
in the column labeled NA. 

Once the checklist has been completed, decisions can be made 
about which elements should be further described by review- 
ing the Items checked NO. 
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Person Completing 

Checklist 

Date 



Appendix 2-A 
PROGRAM ELEMENT CHECKLIST 



YES - Indicates that the written description 
1s adequate 

NO - Indicates Inadequacy, lack of clarity, 
or lack of documentation 

NA - Indicates that this aspect of the element 
1s not relevant 



ADMINISTRATION 



YES 



NO 



NA 



NOTES 



1 , 
2, 
3, 



Procedures for staff 
supervision 



Policies for staff 
evaluation 



Procedures for ensuring 
effective communication 
of Information to staff 



Procedures for ensuring 
effective communication 
of Information to others 



5. 
6. 
7. 
8. 



Hierarchy of reporting 
relationships 



Policies for budget 
authorization 



Procedures for staff 
recruitment and hiring 



Policies for grievance 
and conflict resolution 
for students 



9. Policies for grievance 
and conflict resolution 
for staff 



10, Procedures for long-range 
planning 
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ADMINISTRATION 


YES 


NO 


NA 


NOTES 


11. 


Procedures for short- 
range planning 










12. 


Documentation system 
that supports compliance 
with regulations 










13. 


Procedures to ensure 
health and safety of 
staff 










14. 


Procedures to ensure 
health and safety of 
students 










L5c 


Policies for obtaining 
and upgrading staff 
benefits 










16. 


Administrative roles 
and responsibilities 












STAFFING 


YES 


NO 


NA 


NOTES 


1. 


Job categories 










2. 


Numbers of full and part- 
time staff employed in 
each category 


^ — • 








3. 


Overall student-staff 
ratios 










4. 


Staff-student ratios by 
specific programs 










5. 


Instructional responsi- 
b1 11 ties for each job 
category 










6. 


Administrative responsi- 
bilities for each job 
category 










7. 


Non-instructional respon- 
sibilities for each job 
category 
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STAFF I nu 


VPS 


MO 










8. 


Recordkeeping responsi- 
bilities for each job 
category 














9. 


Evaluation and reporting 
responsibilities for 
each job category 














10. 


Procedures for maintain- 
ing and upgrading of pro- 
fessional skills for each 
job category 










• 




I 1 

II • 


o tuoen u management taSKS 
for each job category 














12. 


Special committee assign- 
ments for each job category 














13. 


Experiential and academic 
preparation of staff by 
each job category 














14. 


Unique requirements of job 
responsibilities and how 
staff characteristics 
match those requirements 














15. 


Special interests and abil- 
ities of staff by each job 
category 














16. 


Procedures for Identifying 
staff training needs 














17. 


In-service activities to 
meet staff training needs 














18. 


Procedures for evaluating 
In-service training activ- 
ities 














19. 


Examples of training needs 
which have been identified 














20. 


Examples of types of In- 
service activities which 
are conducted j 
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BUDGET 



YES I NO I NA 



NOTES 



1. Costs projected by line- 
Item category 



2. Costs projected by general 
purpose 



3. Projection of costs per 
student 



4. Description of costs which 
.are one-time expenditures 



5. Policies for authorizing 
budget allocations 



6. Procedures for planning 
budgets 



7. Procedures for Internal 
fiscal accounting 



Schedule for fiscal re- 
porting 



STUDENT REFERRAL PROCESS 



YES NO NA 



NOTES 



1. Individuals who refer 
students 



2. Procedures by which refer- 
ring Individuals receive 
information about program 
upon which to base a re- 
ferral 



3. Approximate time schedule 
for receipt of referrals 



4„ Content of the referral 



5. Forms which are used in 
the referral process 



STUDENT SELECTION CRITERIA 



YES NO NA 



NOTES 



1. Relevant characteristics 
of the target audience 
as determined by needs 
assessments 
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STUDENT SELECTION CRITERIA 



YES 



NO I NA 



NOTES 



2. Procedures by which a 
pool of eligible stu- 
dents is identified 



3. Procedures by which the 
most educational needy 
within that pool are rank- 
ordered 



4. Selection indicators used 
in the procedures 



5. Means of obtaining in- 
formation for *he indi. 
cators 



6. Way in which indicators 
are combined to identify 
each student's degree of 
need 



7. 
8. 



Cut-off scores used 



Policies for filling open 
slots in the program from 
the waiting list 



9. Exit criteria which specify 
expectations of success or 
mastery 



INSTITUTIONAL GOALS 



YES 



NO NA 



NOTES 



1. Goal and how reaching it 
will enhance the program 



2. Timeline for attaining 
each goal, including var- 
ious tasks to be performed 



3. Individual responsible for 
coordinating activities 
required to reach each 
goal 



4. Other personnel who will 
be involved in the activ- 
ities 
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INSTITUTIONAL GOALS 

5. Procedures for evaluating 
whether the goal has been 
reached 


YES 


NO 


NA 


NOTES 


STUDENT AND STAFF GOALS 

1. Goals categorized by tar- 
get audience 

2. Individual (s) responsible 
for developing new goals 
and revising existing ones 

3. Where written copies of 
all goals are filed 

4. Procedures for using goals 
to plan instructional ob- 
jectives, activities, and 
purchase of commercial 
materials 

5. Procedures for using goals 
to develop individual stu- 
dent plans 

6. Individuals who receive 
copies of goals and how 
this information is used 
by them 

7. Procedures for using goals 
in initial test selection 
activities 

8. Procedures for using goals 
to monitor overall student 
progress 

9. Procedures for using goals 
to monitor staff develop- 
ment 

10. Procedures for using goals 
in evaluation activities 


YES 


NO 


NA 


NOTES 

« 
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STUDENT AND STAFF OBJECTIVES 



YES 



NO 



NA 



NOTES 



Objectives categorized by 
target audience and goals 



Individuals) responsible 
for developing new objec- 
tives and revising exist- 
ing ones 



3. Where written copies of 
all objectives are filed 



4. Procedures for using ob- 
jectives to develop , re- 
vise, or adapt activities 



5. Procedures for using ob- 
jectives to organize con- 
tent and sequence of a 
program or course 



6. Procedures for using ob- 
jectives to develop In- 
dividual student plans 



7. Individuals who receive 
copies of objectives and 
how this Information 1s 
used by them 



8. Procedures for using ob- 
jectives to develop or 
select tests 



9. Procedures for using ob- 
jectives to monitor stu- 
dent progress 



0. Procedures for using ob- 
jectives to coordinate 
activities among courses 
within the Institution 



.1. Procedures for using ob- 
jectives to coordinate 
activities with follow-up 
and work placement out- 
side of the institution 



L2. Procedures for using ob- 
jectives in evaluation 
activities 
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DEVELOPING INDIVIDUAL 
STUDENT PLANS 


YES 


NO 


NA 


NOTES 


1. 


Standard format for each 
plan 










2. 


Where plans are stored 
and how confidentiality 
1s ensured 










3. 


Individual responsible 
for obtaining Information 
to be used 1n the plan 








• 


4. 


Procedures for obtaining 
various types of Informa- 
tion and timeline for 
doing so 










5. 


Policies for when plans 
will be developed and up- 
dated 










6. 


Procedures foi revising 
plans In response to un- 
scheduled occurrences 










7. 


Procedures for revising 
plans In response to 
planned growtn ano oevei- 
opment 










8. 


Individual (s) Involved 1n 
the development of a plan 












INSTRUCTIONAL ACTIVITIES 


YES 


NO 


NA 


NOTES 


1. 


Activities cross-referenced 
to goals and objectives 










2. 


Commercial and staff- 
developed programs 1n use 










3. 


Procedures by which staff 
and students obtain mate- 
Mais 










4. 


Preferred Instructional 
approaches 
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NOTES 



5. Examples o* activities 



6. Procedures for how staff 
work together to develop, 
adapt, and revise activ- 
ities 



7. Procedures for orientation 
of new staff to Instruc- 
tional approaches 



8. Predominant types of 
equipment available 



9. Specific facilities em- 
ployed 



:00RD I NATION AMONG COURSES IN 
THE INSTRUCTIONAL PROGRAM 



YES I NO 



NA 



NOTES 



1. Procedures for ensuring 
coordination 



2. Individual responsible for 
organizing, monitoring, 
and conducting coordina- 
tion activities 



3. Process for Identifying 
Individuals for partici- 
pation 1n coordination 
activities 



4. Schedule of coordination 
activities 



5. Process for updating and 
revising linkages 



j 



COORDINATION WITH FOLLOW-UP 
EDUCATION AND WORK PLACEMENT 



YES I NO 



NA 



NOTES 



1. Procedures for ensuring 
coordination 
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COORDINATION WITH FOLLOW-UP 
EDUCATION AND WORK PLACEMENT 

2. Individual responsible 
within the institution 
for organizing, monitor- 
ing, and conducting co- 
ordination activities 

3. Process for Identifying 
follow-up education and 
work placement programs 
that should be linked to 
Internal programs 

4. Process for Identifying 
and contacting other out- 
side support systems 

5* Schedule of coordination 
activities 

6* Process for updating and 
revising linkages 


YES 


NO 


NA 


NOTES 

« 


STUDENT MANAGEMENT TECHNIQUES 

1. Process by which manage- 
ment techniques are de- 
signed for students 

2. Examples of techniques 
presently being used 

3* Institution-wide standard 
practices 

4. Schedule of periodic re- 
view for effectiveness 
of techniques 


YES 


NO 


NA 


NOTES 1 


NON-INSTRUCTIONAL SERVICES 

1* Goals and objectives or 
statements of purpose 

2. Examples of activities 
and materials 


YES 


NO 


NA 


NOTES 1 
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NOTES 



3. Numbers and types of staff 
Involved 



4. Numbers and characteris- 
tics of students Involved 



5. Procedures for maintain* 
1ng records on non- 
Instructional services 



6. Procedures for evaluating 
non-instructional services 



RECORDKEEPING 



YES I NO 



NA 



NOTES 



1. Types of records main- 
tained and individual (s) 
responsible for collecting 
and updating the necessary 
Information 



2. How the various records 
are used by students , In- 
structors, administrators 
and other program or out- 
side personnel 



3. Samples of recordkeeping 



PROGRAM EVALUATION 



YES I NO 



NA 



NOTES 



1. Purpose of the evaluation 
activities 



2. 
3. 



Key evaluation questions 



Process for management 
of the evaluation 



4. Evaluation design 
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3, DETERMINING THE FOCUS OF EVALUATION ACTIVITIES 



INTRODUCTION 



Evaluation activities should be focused first by clarifying 
the purposes of the evaluation and then by Identifying th^ 
specific program elements to be addressed. Most programs 
will have neither the financial resources nor the staff time 
to conduct yearly evaluations of all of their elements. The 
problem, then, 1s to determine where to apply the resources 
available for evaluation pur poses. Obviously, the evalua- 
tlon activities required by Federal, state and local agen- 
cies must be conducted; but what other aspects of a program 
should be evaluated? How 1s a determination made as to the 
placement of resources necessary to obtain the most useful 
Information for program change and improvement? 

There are two steps which, if followed, will help to deter- 
mine where to best use resources and to guide in the 
development of an evaluation plan. First, program elements 
of Importance should be identified, Then v specific evalua- 
tion questions should be developed for each program element 
of Interest, This section presents a process for accomplish- 
ing these two steps. The results of this process will be one 
or more well-focused evaluation questions -- questions 
which, when answered, will provide the staff with informa- 
tion for program i improvement • 



SELECTING PROGRAM ELEMENTS FOR EVALUATION PURPOSES 



In order to select elements to be evaluat d, the key program 
elements must first be Identified. Complete descriptions of 
elements, such as those discussed in Section 2, are a refer- 
ence point for this identification. The choice of which 
elements to evaluate can then be made on an informal basis, 
by Identifying those of greatest concern or those about 
which little 1s known. Or, If it 1s unclear as to which 
program elements should be evaluated, a more systematic 
process can be used- to. help guiJe. the determination. One 
such brief process which will result in an ordered list of 
elements 1s as follows. 
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With the existing program description as a guide, a 11st of 
program elements can be developed. Each element should be 
paraphrased clearly and succinctly and then rated by asking 
the four questions provided. Each response should then be 
scored as suggested below. 



Question 1 ; Has this element been evaluated before? 

NO - score 2 (two) points 
SOMEWHAT - score 1 (one) point 
YES - score 0 (zero) points 

Question 2 : Will evaluating this element provide informa- 
tion that will help to make decisions or 
policies? 

YES - score 2 (two) points 
MAYBE - score 1 (one) point 
PROBABLY NOT - score 0 (zero) points 

Question 3 : Has this element been of concern or problematic 
1n some way? 

VERY MUCH - score 2 (two) points 
SOMEWHAT - score 1 (one) point 
NOT MUCH - score 0 (zero) points 

Question 4 : Have there been external requests for informa- 
tion about this element? 



YES, DIRECT REQUESTS - score 2 (two) points 
YES, BUT INDIRECTLY OR IMPLIED - score 1 

(one) point 
NONE - score 0 (zero) points 



Now, for each element, the score for each of the four ques- 
tions 1s added up to arrive at a total score for that ele- 
ment. This number 1s then used to rank order the elements. 
Those with the highest score are probably most relevant for 
the present evaluation purposes. 



Following this process will result in an ordered list of 
program elements -- a 11st which indicates where best to use 
the resources available for evaluation activities. Rather 
than using limited resources to attempt to evaluate every- 
thing, which often results In superficial Information, it is 
better to conduct a thorough evaluation of one or two 
program elements during the course of the year and then move 
on to other program elements in following years. 
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Appendix 3-A includes a worksheet which can be used to order 
program elements, following the steps and process just de- 
scribed. Also included is a brief example showing how this 
procedure worked for one program. 



DETERMINING EVALUATION QUESTIONS 



After the program elements have been selected for evaluation 
purposes, it is necessary to determine the questions which 
will be answered. Developing specific questions wi 1.1 help 
focus the evaluation activities on producing useful i nf orma- 
tion which directly addresses the actual topics of concern. 



Why Evaluation Questions Are Necessary 

Consider briefly what might happen if specific questions are 
not developed before conducting the evaluation activities. 
Take a situation, for example, where the decision has been 
made to evaluate students' cognitive achievement through the 
use of a standardized achievement test. After pre* and 
posttests are administered, the students' average gains are 
calculated. The results are then presented to the members 
of an advisory committee and the program instructors. 
Unfortunately, however, during the meeting the committee 
members say, M But what we re ally want to know is: What 
specific skills have been mastered? . . . How many students 
actually mastered each skill?" The instructors, on the other 
hand, ask: "What specific skills are students not mastering? 
In what areas does instruction need to be improved?" 

Some of these unanswered questions may be answered by re* 
scoring tests or reanalyzing data, but this would require 
additional staff time. Or, the answers to these questions 
might not be available at all because the test used did not 
collect the necessary information. In any case, by not first 
determining a specific evaluation question or set of ques- 
tions, the results might be unsuitable for meeting the real 
needs of those using the information. 

This need for determining the questions becomes even more 
critical when evaluation activities move further away from 
the use of the conventional achievement test and more toward 
the collection of information for program implementation or 
process evaluation purposes. For example, in a situation 



54 

3-3 



where one of the major program elements to be evaluated is 
that of Instructor-student Interactions, what types of 
Interactions should be measured: verbal, non-verbal, or 
both? Should evaluation activities focus on Interactions 
oriented to group management, Interpersonal relationships, 
direct instruction, or all three? These types of decisions 
depend on what the staff and policymakers want to know. 

The possibilities for uncertainty regarding what to focus on 
multiply rapidly 1n less conventional evaluation areas. 
These uncertainties, such as choosing the type of Instrumen- 
tation, data col lection techniques, ana lysis procedures and 
so forth, need to be resolved as much as possible before 
beginning the evaluation activities. Having well specified 
evaluation questions before beginning any activities helps 
ensure that the evaluation will proceed without wasting 
time, that Information will not have to be collected again, 
and that the results will be useful for program evaluation 
purposes . 

Developing the Evaluation Questions 

After one or more program elements have been selected for 
the evaluation activities, it 1s time to develop the actual 
questions. Each element to be evaluated should be con- 
sidered one at a time 1n order to develop a list of ques- 
tions. These questions should be ones whose answers will 
provide Information on the extent to which this element is 
effective or on how successfully it has been implemented. 

For example, 1f the evaluation is focusing on the remedial 
mathematics element of a program, some questions whose 
answers may help evaluate that element might include: 

1. At what level are the students 1 math skills on 
entering the program, 1n comparison with their 
peers not in the program? 

2. How does student participation in math 
activities compare with participation in 
reading activities? 

3. To what extent do the instructors believe that 
the program is effective for students' math 
lea rn 1 ng? 

4. To what extent do the instructors think that 
students are appropriately selected for the 
program? 
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5. Do instructors diagnose student needs accu- 
rately? 

6. Do instructors prescribe instructional objec- 
tives and learning activities which directly 
address student needs? 

7. Does the program contribute to students 1 doing 
we 11 in later programs? 

The answers to each of these questions could provide 
information useful for determining the success of the 
program or the leve.l of program implementation. 

There is no formula for ensuring that the evaluation 
questions developed are the best ones. However, there are 
three guidelines which, if followed, will make it more 
likely that the questions asked will provide useful evalu- 
ation Information in an efficient manner. These guidelines 
are clarity, relevance and utility. 

For a question to follow the guideline of clarity, the 
meaning of that question must be clear to those who read the 
question; they should be able to agree on what the question 
means. There are two areas which are often unclear and thus 
special attention should be paid to them when writing the 
questions: (1) the object to be measured, and (2) the 
standard or comparison for that measure. With most evalua- 
tion questions there will be a standard or comparison, but 
1n some situations the question may only describe something; 
thus there will not be a standard or comparison. 

Consider the Issue of clarity 1n regard to the evaluation 
question "How effective Is the math component?" What is the 
object that will be measured? What standard will be used to 
measure this effectiveness? This evaluation question is too 
vague and thus could mean many things to different readers. 
In comparison, consider the second question in the list 
previously given: "How does student participation in math 
activities compare with participation in reading activi- 
ties?" It 1s fairly clear that the objects to be measured 
are those of student participation in math and reading 
activities. Further, 1n this case a comparison of the level 
of participation 1n math activities to the level of partici- 
pation 1n reading activities will be used to determine 
ef f ect i veness . 

Consider briefly the fifth questio., in the list: "Do 
Instructors diagnose student needs accurately?" The object 
to be measured here 1s how well the instructors do at 
diagnosing student needs. To answer this question it would 
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be necessary to devise some measure of instructors 1 abili- 
ties to diagnose those needs, perhaps by having each 
instructor prepare a written documentation of a diagnosis 
for a particular student. A model diagnosis could be 
developed to act as the standard, with another person making 
a judgment as to how well each instructor had diagnosed the 
needs. By comparing instructors' diagnoses with the model 
it would be possible to determine how well the teachers 
actually do at diagnosing student needs. 

For an evaluation question to follow the guideline of .rele- 
vance, each question should (1) contribute information use- 
ful in fulfilling the intended purpose of the evaluation 
activities, and (2) provide useful information to those who 
determine program policy and to those who implement the pro- 
gram. It is important that the purpose of the evaluation 
activities, as discussed in Section 1, should be a 1 ready 
identified before this point. Questions which will not 
provide information toward the intended purpose, although 
they may be very interesting, should probably not be pursued 
unless resources are unlimited. Appendix 3-B includes a 
list of evaluation purposes, as discussed in Section 1,. 
along with some sample evaluation questions. These questions 
are provided only as examples and certainly do not cover the 
full range of evaluation questions that may be asked in 
relation to a specific project. 

For the guideline of relevance to be fully met, the ques- 
tions should also provide useful information to those who 
actually implement the program. The questions asked should 
provide information to the project director, the members of 
the advisory board, the program staff, and others involved 
with the program. The best way to ensure that the questions 
are relevant is to involve these people in formulating the 
questions during the planning activities for the evaluation. 

For an evaluation question to follow the guideline of util- 
ity, there [nust be a projection about whether those with 
appropriate authority will really use, or be open to using, 
the results. For example, consider the evaluation question: 
"To what extent are the goals of the program sufficiently 
focused to guide instructional, supportive, and student 
assessment activities?" If it is already known that a Board 
member, such as a State Education Department staffer, and 
the Program Director already have concerns about this issue, 
then they are likely to be especially attuned to answers to 
the question. They will be more likely to consider the in- 
formation seriously in changing policies. Therefore, if the 
question is also clear and relevant, it should get h ; g h 
priority for evaluation activities. 
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In comparison, consider the evaluation question: "In what 
ways do the interactions of the Project Director and the 
staff affect the instructional activities?" If those 
planning the evaluation realize that the Project Director 
may have problems with the staff* is very resistant to 
criticism or change, and is secure in his or her position, 
then a.nswers to these types of questions will probably not 
be used constructively. Therefore, the question is probably 
not a very practical one and might be eliminated; more 
constructive or useful questions should probably be asked. 

Once potential evaluation questions have been re vi ewed for 
clarity, relevance and utility, a final concept should be 
addressed: the questions should cover a broad ra-nge of 
aspects for the element being evaluated. In other words, 
the questions should not focus on a very narrow aspect of 
the program element, such as only attendance data or only 
cognitive outcomes. There is a broad range of possible 
questions for each element and this range should be fully 
covered. 



Consider the question 6\ scussed previ ous ly : "How does 
student participation in math activities compare with 
participation in reading activities?" One might simply 
collect basic factual information on who did or did not 
participate in math and reading activities. Or participa- 
tion could be dealt with on another level where some 
interpretation by the instructor or another observer must be 
made about the extent of a student's involvement in an 
activity beyond mere presence. This would be at a different 
level than the attendance, i the sense that it addresses 
more complex patterns of behavior. The resulting informa- 
tion would probably provide more useful information for 
program improvement than would a simple record of atten- 
dance. 



Question 5, "Do instructors diagnose needs accurately?" is 
another example of the type of question that requires more 
than the basic recording of factual information, A judgment 
is required about a fairly complex pattern of behavior on 
the part of the instructor. Finally, consider the question: 
"Does the program contribute to students 1 doing well in 
follow-up programs?" This question moves the evaluation 
activities to another level: determining whether the 
program helps students in follow-up placements. 

It should not be implied that some types of questions are 
better than others for evaluation purposes. In fact, 
sometimes the answers to basic factual questions must be 
collected in order to interpret answers to other questions. 
The point here is- simply - that- the evaluat i on questions 
should focus on a variety of types of questions and not just 
on one type. 
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ORDERING PROGRAM ELEMENTS - WORKSHEET 



Evaluated Before 
No - 2 

Somewhat • 1 
Yes - 0 



ELEMENT 




u 
I 

oo 



Information Mil 
Help Decisions/ 

Policies 
Yes - 2 
Maybe - 1 
Probably Not - 0 






Problematic/High 
Concern Area: 
Very Much - 2 
Somewhat - 1 
Not Much • 0 



External Request 
For Information: 
Yes, Direct - 2 
Yes, Indirect * 1 
None - 0 
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ORDERING PROGRAM ELEMENTS - EXAMPLE 



ELEMENT 



Math 
Tutorlnt 




w 

I 

10 
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Micro- 
Computer 
Reading 
Software 

Student 

Selection 

Procedur es 

Teacher 
Diagnoses 
and Pre- 
scriptions 



Evaluated Before 
No - 2 

Somewhat • 1 
Yes - 0 



Information Will 
Help Decisions/ 

Policies 
Yes - 2 
Maybe - 1 
Probably Not - 0 



Problematic/High 
Concern Area: 
Very Much - 2 
Somewhat - 1 
Not Much - 0 



External Request 
For Information* 
Yes, Direct - 2 
Yes, Indirect - 1 
None - 0 
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In this example, an ordered list of elements would be: 

Student Selection Procedures (7) 
Math Tutoring (5) 

Microcomputer Reading Software and Teacher Diagnoses 
and Prescriptions tied (4) 

Student selection procedures is the highest ranked element 
and thus probably the most important to evaluate at tills 
time, As resources permit, evaluation activities could be 
conducted on Math Tutoring, then the remaining elements, as 
indicated by their rank order, 
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Appendix 3-8 
SAMPLE EVALUATION QUESTIONS 



Purpose : Accountability and Reporting 

1. Is the program in compliance with all of the 
funding agency 1 s regulations and requirements? 

2. How could reporting to funding agency be improved? 

3. Are adequate records being kept of: 

demographic in formation about students? 

student participation? 

attendance? 

materials acquisition and use? 
amount of instruction? 
behavior/disciplinary problems? 
length of participation? 
student needs and student progress? 

4. Has needs assessment data been used to plan programs? 

5. Is the program serving the students which 1t was in- 
tended to serve? 

6. On what basis have participants been selected? 

7. Is staffing adequate to achieve program objectives? 

8. Is data being collected to assess the impact of the 
program? 

9. What are the relative costs of different program com- 
ponents? 

10. How are costs related to program priorities? 

11. What is the average cost 

per student? 

per hour of instruction? 
per unit of achievement? 

12. How do costs vary by type of program? 
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Appendix 3-B (Continued) 
Purpose ; Assessing Short-Term Effects of Programs 
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1. What specific skills did students learn as a result of 
program participation? 

2. How many skills do students master per unit time in the 
program? 

3. Do students Improve their performance on criterion- 
referenced tests as a result of participation in the 
program? 

4. As a result of participation in the program, do 
students Improve their performance on norm-referenced 
tests? 

5. Do students change their attitudes about: 

the program? . 
school ? 
learning? 
self? 

control over their lives? 

reading? 

math? 

emp 1 oyment opportunities? 
future educatl on? 
the Institution? 

6. Co students 1n the program learn significantly mo re 
than students who did not participate in the program? 

7. Do students show changes in their classroom behavior? 

conf 1 dence? 
coopera t 1 veness? 
study habits? 

Interaction with teachers? 
Independent study? 
t 1me-on-task? 

8. How do non -1 n s t ruct i ona 1 services affect academic 
gains? 

9. Do some types of students show greater gains than 
others? 
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Appendix 3-B (Continued) 
Assessing Long-Term Effects of 



Programs 



Do some types of programs or methods produce longer 
term effects than others? 

Do recidivism rates vary as a function of 

program participation? 
achievement level? 
attitude? 

amount and types of services received? 
other related issues? 

jfrogram participants more successful at finding 
What kinds of students are most successful at 
jobs? 

do skills acquired in the program relate to the 
./ass of iobs students find? 

Do a higher proportion of program participants return 
to school after release? 

What kinds of students are most likely to return to 
school ? 

What skills do former program participants find 
themselves using most? 

life skills? 
job skills? 
academic skills? 
interpersonal ski 1 Is? 

In retrospect, how do former participants view 
strengths and weaknesses in the program? 

What kinds of problems do former program participants 
face that might suggest changes in the program? 

How do measures of cognitive gain vary across several 
years of program operation? 

What changes have taken place in the program since its 
inception and what impact have these changes had on its 
ef f ecti veness? 
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Appendix 3-B (Continued) 



Purpose : Determining the Degrees to Which a Program Has 
Been Implemented ~~ 



1. Does the program have formal written objectives? 

« 

2. Are program objectives realistic given constraints 
within the institution? 

3* What are the key elements of the program and how many 
are 1n place? 

4. In what different ways has each key element of the pro- 
gram been 1mpl emented? 

5. To what extent do concerns about management routines 
(scheduling, location of materials, etc.) exist among 
staff? 

6. To what extent do concerns about altering the program 
to better meet students' needs exist among staff? 

7. Are classroom activities consistent with the objectives 
of the program? 

8. What portion of Instruction time 1s directly spent on 
tasks related to objectives of the program? 

9. Do the materials being used match the objectives of the 
program? 

10. What materials and activities have been most often 
used? 

11. Is the mode of teacher-student interaction consistent 
with program objectives? 

12. Is the program serving the students it was intended to 
serve? 

13. Is information about the operation of the program being 
used to improve the program? 
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Appendix 3-B (Continued) 



Purpose ; Management and Staff Effectiveness 

1. Are some teachers more effective than others with cer- 
tain types of students? How can this differential 
effectiveness be used to improve instruction? 

2. Are some teachers more effective than others in. using 
certain types of materials or techniques? 

3. In what ways is the administration of the institution 
being helpful or obstructive? 

4. What kinds of In-service training programs have been or 
would be most useful to program staff? 

5. How might improved management practices improve the 
quality of the program? 

6. How might communication among staff members be im- 
proved? 

7. How could roles and responsibilities be better defined? 



Purpose : Determining How Best to Match Services with Indi 
vldual Studert's Needs 



1. What are the predominant academic problems of students? 

2. Does the nature of predominant problems vary by age of 
student? 

by socio-economic status? 
by program component? 

3. What types of learning activities a re effective for 
which types of problems? for students with different 
learning styles? 

4. Do program activities accommodate a broad range of 
skill levels? Is there a sufficient variety of mater- 
ials and learning activities for each skill level? 
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Appendix 3-B (Continued) 



5, How well do students fe>l that the program is matching 
their needs? What Improvements could be made? 

6. Which component of the student needs assessment system 
is most useful to program personnel? Which components 
might be dropped or modified? 

1. How effectively are Individual student plans used to 
guide instruction (1.e, f selection of materials and 
techniques)? 

8. How well are non-cognitive needs being met? 



Purpose : Identification of Relationships Among Services and 
Program Components ~ 

1. To what extent does this program supplement other pro- 
grams? 

2. In what ways could this program be better coordinated 
with other programs? 

3. How do conflicts with other programs or activities af- 
fect participation 1n this program and how might they 
be overcome or minimized? 

4. How does participation in this program affect perform- 
ance or participation in other programs? 

5. What factors influence a student's choice of services 
(given the option to choose)? 

6. How much 1s Information about students shared across 
programs? 

7. What 1s the pattern of student referrals by one service 
component to others? 

8. What services did students receive as a result of re- 
ferrals from other program components? 
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Appendix 3-B (Continued) 



9. What are service providers 1 perceptions of the effec- 
tiveness of other components? How accurate are those 
percept i ons? 

10. -What are service providers' perceptions of the service/ 

roles of other components? How accurate are those per- 
ceptions ? 

11. How might communication among programs be facilitated? 



Purpose : Identification and Description of Effective Prac - 
tices 

1. How can data be collected to identify practices which 
lend themsel ves to 

positive attitude change? 
cognitive gains? 
behaviora 1 change? 
long-term program effects? 

2. Are some techniques viewed more positively than others 
by teachers? ... by students? ... by adminis- 
trators? 

3. Do techniques which are viewed most positively by stu- 
dents and/or teachers yield the best results? 

4. How could staff make better use of effective practices 
(techniques and materials) from other institutions? 

5. How could practices be disseminated to other institu- 
tions? 

6. Are techniques differentially effective with different 
kinds of students (ages, sexes, types of offense, 
etc.)? 
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4. STRATEGIES AND TECHNIQUES FOR PROGRAM EVALUATION 



INTRODUCTION 



There are many strategies and techniques which can be used 
to evaluate a program, depending upon the purpose of the 
evaluation activities and based on the particular prefer- 
ences held by those planning the evaluation. For jus.t this 
reason and because there are no comprehensive evaluation 
models for Chapter 1 Neglected or Delinquent programs, this 
section Is designed to present some basic approaches which 
can be used to evaluate selected program elements in order 
to provide information for reporting requirements and pro- 
ram improvement purposes. 

It is recognized that interest in, and resources for, evalu- 
ation activities varies with the individual N or D program. 
Some programs a re content to simply report descriptive data 
about their projects while others have attempted more ambi- 
tious evaluations designed to show the impact of the ser- 
vices and to determine where improvements might be made. 
Even among those N or D programs interested in conducting 
evaluations, there are basic philosophical differences about 
what strategies and techniques are appropriate given the 
unique conditions under which N or D services are often pro- 
vided and the characteristics of the clients being served. 
Some argue that any evaluation of an N or D program has to 
be grounded in a standa rdi zed norm-referenced or criterion- 
referenced test that will yield student achievement data. 
Others feel that such test data is inappropriate for evalua- 
ting the N or D program and prefer to emphasize alternative 
app roaches , 

This section will not attempt to resolve the philosophical 
differences that exist over what evaluation strategies or 
techniques are best suited for N or D programs. Rather, it 
will present some approaches and techniques that can be used 
to collect information necessary to answer the evaluation 
questions of interest. The choice of approaches and tech- 
niques must be made in view of the evaluation questions be- 
ing asked and is best left up to those responsible for the 
evaluation activities. It should also be noted that this 
section is a general overview of the selected strategies and 
techniques. Some are relatively straightforward and, depend- 
ing upon the. expertise- of those conducting the evaluation 
activities, easily implemented. Others are more complex and 
might require further training or outside expertise in order 
to be implemented. 
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Assistance in planning N or D evaluations or training in any 
of these techniques can be obtained by calling a Regional 
Chapter 1 Technical Assistance Center. 



TESTING APPROACHES 



One fairly common approach used to evaluate the effect of a 
program is that of testing. In very gen^rc! terms, some type 
of achievement test is administered i i: order to measure 
changes m student performance. If t>'« improvement in per- 
formance is greater than would have b±sn expected without 
the program, then the program is judged to have had a posi- 
tive effect. So, tests might be used to collect information 
to answer the evaluation questions: "Do students 1n the pro- 
gram learn significantly more than student* not in ttie pro- 
gram?" or "What specific skills did students l~:-n * s a 
result of participation in the program?" 

This section will deal with the application of both ru- fa- 
referenced and c r i t e r i on -ref e renced testing approaches for 
program evaluation purposes. A variety of testing applica- 
tions is presented, as well as possible constraints on using 
these approaches in the N or D setting. Information on spe- 
cific tests is provided in Section 5. 



Norm-Referenced Testing 

A norm-referenced test is a test that is used to determine 
an individual's status with respect to the performance of 
other individuals on that test. This definition implies that 
the purpose of a norm- referenced test is to compare the per- 
formance of an individual with that of others. When standar- 
dized norm-referenced tests are used in a school setting, 
this comparison is usually made w'th those of a similar age 
or grade level. The performance of the comparison group is 
found in a table of norms which have been derived through 
previous administrations of the test to selected school sam- 
ples . 



Problems in Using Norm-Referenced Tests in N or 0 Evalua - 
tions . There are a variety of potential problems associated 
with using norm-referenced tests in N or D settings. Before 
making decisions regarding the use of a norm-referenced test 
as part of the evaluation activities, these problems should 
be considered. 7 7 



The published norms are usually inappropriate 
for use with an N or D population. The group 
from which the norms are derived generally 
represents an average school population. Since 
an N or D group is atypical and may not per- 
form like a typical school group, comparisons 
made with such a group may be difficult to 
i nt e rp ret . 

In order for test scores to be comparable with 
the norms, the test should be administered 
within specified testing periods during the 
year. As students are continually entering ar>d 
leaving the N or D setting and staying for 
varying periods of time, it is often imprac- 
tical to restrict the testing to these speci- 
fied dates. 

The test is supposed to b l* administered in a 
precisely defined way. The mode of operation 
of an N or D institution and the characteris- 
tics of its students often make it very dif- 
ficult to follow the standardized testing 
procedures . 

The N or D students may no; always be motiva- 
ted to do their best on a norm-referenced 
test. In addition, for a variety of reasons, 
students will often exit from the program 
without receiving a posttest. 

Norm -referenced tests a re general ly meant to 
serve many different programs and a variety of 
populations; therefore their content may be 
too broad to adequately test the specific 
skills covered in t* 4 : N or D program. As a 
result, the amount of real improvement may be 
underestimated by these tests. A norm- 
referenced test will be especially insensitive 
to the small improvements made by the many N 
or D students who are in the program for only 
a brief period of time. 

The items on a norm-referenced test are typi- 
cally selected to spread out the range ot 
scores so that individual-to-group comparisons 
will be facilitated. This wide range of item- 
often makes it difficult to relate a student's 
score to specific instructional needs. 

The language used in norm-referenced tests is 
not always appropriate for N or D students. 
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The skill deficiencies of the \i or D student 
will sometimes require the administration of a 
test that is considerably below the student's 
grade level. Here the student encounters lan- 
guage and situations aimed at much younger 
students, and resentment may aevelop. Some 
tests have now appeared which attempt to over- 
come this problem by using a high- interest 
low-ability approach. 



Ways to Use Norm-Referenced Tests in N or D Evaluations . 
Four ways of using norm-referenced tests are described in 
this section. The first concerns the determination of cur- 
rent status using the norms tables, while the other three 
pertain to the determination of overall program effect. Any 
method chosen should be appropriate to the particular pro- 
gram being evaluated and should provide information bearing 
on the evaluation questions at issue. In many cases it may 
be apparent that the use of criterion-referenced tests, to 
be described later, will yield more useful information. 

1. The Use of No rm- Re f e re n ce d Tests to Determine Current 
Status -- Sometimes it may be of interest to determine 
how students 1 performances in an N or D program compare 
with those of a typical school population. The types of 
evaluation questions being asked might include: "How far 
behind their public school peers is this group of N or D 
students?" or "In which subject areas do the N or D stu- 
dents need special work?" For either question, norm- 
referenced testing could be appropriate. Similarly, if 
the evaluation activities are focusing on the element of 
student selection and the question is: "Which students 
are most in need of participation in the program?", 
norm-referenced testing could again be appropriate. 

When using norms to determine current status it is im- 
portant to test the students within the dates for which 
the norms are established and to adhere as closely as 
possible to the standardized testing procedures. It must 
be remembered that the comparison is being made with a 
different population of students and any interpretation 
of the results should take this into account. 

2. The Use of Norm-Referenced Tests to Determine Overall 
Effect -- Here the N or D students are given both a pre- 
test and posttest. The position of these students rela- 
tive to the norming group is determined both at pretest 
time and at posttest time. Any improvement in their po- 
sition is assumed to be due to the special educational 
treatment being provided in the program. Generally, the 
types of evaluation questions being asked might include: 
"Do students in the program learn significantly more 
than students who did not participate in the program? 1 ' 
or "How do measures of cognitive gain vary across sev- 
eral years of program operation?" 
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Both the pretest and the posttest must be administered 
within the testing periods for which norms are estab- 
lished, regardless of when students enter or leave the 
program. Normlng periods generally occur near the begin- 
ning or toward the end of the school year and, for some 
tests, the middle of the year as well. Thus students 
would have to be in the program for several months in 
order to receive both the pretest and posttest. Here 
again, the standardized testing procedures should be 
followed as closely as possible and interpretations 
should take into account the fact that comparisons are 
being made with a different population of students. 

The Use of Norm -Referenced Raw Scores and Standard 
Scores It is possible to obtain a measure of overall 
program effect without ref e rri ng. to the published norms. 
Here again, the N or D students are given both a -pretest 
and a posttest, but it is the change in their raw scores 
or standard scores that is used to determine program ef- 
fect. Where a choice exists, the standard scores should 
be used since their statistical properties are more 
amen ab 1 e to the calculation of score changes. The types 
of evaluation questions being asked might include: "Do 
students, improve their performances on the test as a 
result of participation in the program?" or "Do students 
need more of an instructional emphasis on a particular 
subject matter?" 

As adherence to specified testing periods is not re- 
quired with this approach, students can be pretested and 
posttested upon entry into and exit from the program. In 
addition, students with only brief stays . in the program 
can now be included in the analysis. 

The major prob'^m with this approach is the difficulty 
of interpreting the meaning of a particular gain, wheth- 
er in raw score or standard score units, without refer- 
ence to norming information. To say that an N or D group 
has gained nine standard score points does not convey a 
great deal without knowing what others have done. One 
way of dealing- with this problem has been to convert raw 
scores to grade equivalent scores and to express gains 
in terms of grade equivalents. This approach is defi- 
nitely n ot recommended due to the misinterpretations 
associated with the use of grade-equivalent scores. 

The Use of the Systematic Allocation Model This model 
can be used to evaluate an educational program within an 
N or D setting if there exists within the setting stu- 
dents who are not. in the program and, if students are se- 
lected for the program on the basis of need. The type of 
evaluation questions asked might include: "Do students 
in the program demonstrate a greater level of academic 
improvement than those not in the program?" 
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It should be noted that the Systematic Allocation Model 
can also be used with c ri te ri on - re f e re n c e d tests. Ap- 
plication of this model with a criterion-referenced test 
eliminates the problems associated with norm-referenced 
testing and the N or D program. (For a detailed discus- 
sion of the Systematic Allocation Model see the Handbook 
for Eva luation of T i 1 1 e I Programs in State Insti tutions 
Tor Neg lecte? cr Delinquent Youth , 1978. This Handbook 
was developed by the System Development Corporation, 
under U.S.O.E. contract number 300-76-0093.) 



Criterion-Referenced Testing 

A criterion-referenced test is a test that is used to deter- 
mine an individual's status with respect to specified objec- 
tives of instruction. For each objective assessed in the 
test, a set of items is developed tc determine whether the 
student has, in fact, mastered that objective. Prior to the 
administration of the test, a criterion is established which 
is then used as the standard to determine whether the re- 
sulting scores indicate mastery or non-mastery of each ob- 
ject i ve tes ted . A c r i t e r i on - ref e renced test may cover one 
or more instructional objectives, depending upon the purpose 
of the test. 

In deciding whether to use n o rm - re f e ren ce d or criterion- 
referenced tests for evaluation purposes, the types of 
information that each will provide should be taken into con- 
sideration. With criterion-referenced testing, each stu- 
dent's score is compared to the prespecified standard to 
determine mastery, while in norm-referenced testing the stu- 
dent's score is compared to that of the norming group. This 
means that in c r i t e ri on - ref e ren ced testing the resulting 
scores indicate which instructional objectives have been 
mastered, while in no rm -referenced testing the resulting 
scores indicate the student's position relative to the 
scores of the norming group. 

Typically, in norm-referenced testing the focus is on indi- 
vidual-to-group comparisons, test content is general and may 
not be matched to a particular instructional content, and 
items are selected to deliberately spread out the score dis- 
tributions. With criterion-referenced testing, the focus is 
on individual comparisons to prespecified standards, the 
test content is much more specific and thus easier to relate 
to a particular instructional content, and the items are not 
selected to spread out the range of student scores. 



Uses of Cri teri on-Ref erenced Testing for Program Purposes . 
There are many ways to use criterion-referenced tests in 
order to collect information useful both for student assess- 
ment and for the evaluation of the N or D program* Four gen- 
eral applications are described briefly and should be kept 
in miad when making testing decisions. 

1. Criterion-referenced tests can be used o as- 
sess the strengths and weaknesses of indi- 
vidual students as they enter the educational 
program. Information regarding the degree to 
which various instructional objectives have 
already been mastered will help in developing 
the student's individual plan. This same i n - 
formation can also be used to plan and evalu- 
ate the instructional programs. 

2. Criterion-referenced tests can be used to as- 
sess the status of individual students as they 
complete segments of the instructional pro- 
gram. This information can then be used to 
determine whether additional instruction is 
required in that area or whether the student 
is ready to move on to a new area of instruc- 
tion. This same information can also be used 
to evaluate the effectiveness of the instruc- 
tional program and the curriculum materials 
being used within the program. 

3. Criterion-referenced tests can be used to as- 
sess the strengths and weaknesses of a group 
in order to determine where to place emphasis 
within the overall i ns t ruct i pna 1 program. This 
type of information can be used in program 
development, grouping, staff assignments, and 
in the evaluation of the later success of the 
instructional interventions. 

4. Criterion-referenced tests can be used to de- 
termine the overall effect of an educational 
program. The information obtained can be used 
to describe program effects and as a planning 
guide for possible program improvements. The 
processes involved in this type of evaluation 
are discussed in the following section. 
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Ways to Use Criterion-Referenced Tests in Evaluating Overall 
Program Effect . Two ways of using cri terion-ref erpncpd taac 
for evaluating program effects are discussed in this sec- 
tion. Both methods have the advantage of not being restric- 
ted by the problems u-sually associated with norm-referenced 
testing, and both will provide relevant information for 
evaluation purposes. 
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The Use of the Systematic ''location Model As dis- 
cussed earlier, this model r.n be used to evaluate an 
educational program within an N or D setting if there 
are students within that setting who are not in the pro- 
gram and if students are selected for the program on the 
basis of predetermined index of need. The types of eval- 
uation questions that might be asked include: "What spe- 
cific skills did students learn as a result of program 
participation?" or "Do students in the program learn 
significantly more than students who did not parti: -pate 
in the program?" 

Application of this model has already been described in 
the section on norm-referenced tests. The model is ap- 
plied in exactly the same way when criterion-referenced 
tests are used. However, the use of criterion-referenced 
tests has the advantage of allowing for the simultaneous 
application of the Criterion Model, as described next. 

The Use of the Criterion Model -- This model can be used 
to evaluate an educational program within an N or D set- 
ting when the type of comparison group required in the 
Systematic Allocation Model is not available. This would 
be the case either when all or practically all students 
are assigned to the program, or when assignment to the 
program cannot be based on the cutoff score used as the 
index of need. The types of evaluation questions that 
might be asked include: "How many basic reading skills 
do the students master during the first month of the 
program?" or "Which instructional objectives are still 
not mastered by students after leaving the program?" 

The Criterion Model requires that a performance standard 
be set in advance for the c r i te r i on - ref e ren ce d test be- 
ing administered to the students in the program. That is 
to say, what the group is expected to accomplish in 
terms of mastery on the test must be stated in advance. 
Criteria may be established in a variety of ways, such 
as based on prior performance of a similar group or by 
teacher judgment as to what should be expected. Indi- 
rectly then, expectations regarding the group's level of 
mastery on the objectives being tested must be set in 
advance. For example, in a basic skills mathematics 
program, and with reference to a particular criterion- 
referenced test, it may be decided that 80% of the stu- 
dents in the program should be able to score 75% correct 
on the addition and subtraction items and 70% correct on 
the multiplication and division items. Students are pre- 
tested with the criterion-referenced test as they enter 
the program and posttested with the same test as they 
leave the program. The posttest results are com pared 
with the pre-established standard to determine which of 
the criteria h.ave or have not been met. 



77 



Pretest and posttest results are compared to determine 
what kinds of improvements have taken place in the stu- 
dents' levels of performance. Information thus collected 
can provide input toward program change and improvement. 

It should be noted that the Criterion Model does not 
provide a definitive evaluation of the educational pro- 
gram, since no comparison group is involved. The results 
observed could be due to other instruction received or 
to outside causes. Nevertheless, the information pro- 
vided by application of the model is suggestive of pro- 
gram effect and surely provides direction for program 
evaluationandimprovement. 

N or D programs which are able to use the Systematic 
Allocation Model can simultaneously apply the Criterion 
Model and thus obtain the additional i n format i on this 
model provides. In order for this to be done, of course, 
a criterion-referenced test must be used when implement- 
ing theSystematic Allocation Model. (For a more de- 
tailed discussion of either model see the Handbook for 
Eva 1 uati on of Title ± Programs in State Institutions Tor 
Neg lectea o_r Delinquent Youth / 19787]T 

Constructing a Criterion-Referenced Test . Sometimes it is 
necessary to construct a criterion-referenced test for the 
purposes of the N or D evaluation. In general, this is rec- 
ommended only when evaluating the effect of a short unit of 
instruction. When evaluating the overall effect of an in- 
structional program 1t 1s better, if possible, to use a pub- 
lished criterion-referenced test. This is because the con- 
struction of a statistically sound criterion-referenced test 
which is approp-late for evaluating an enti re -p rog ram is an 
involved proposition which requires much in the *ay of time 
r.nd resources. It therefore is advisable to initially review 
ti:e available published tests to determine whether they meet 
tl»e evaluation needs. 

In developing criterion-referenced tests for use in evalua- 
ting individual units of instruction, the following steps 
arG recommended. The steps are only briefly described here 
in order to provide an overview of the process involved. 

1- Selectthe objectives to be measured . These 
objectives should be ones taken from the in- 
structional program being implemented or to be 
Implemented. The objectives should be stated 
1n measurable terms, clearly indicating the 
expected learning outcome of the student. 
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Develo p tt j test specifications . Specify what 
the tc * s going to loo - k like, Determine how 
many 'ttms will be included for each objec- 
tive, the it em format(s) to be used (e.g., 
multiple choice, true-false), the reading lev- 
el, vocabulary, organization of the items, and 
how the student will respond (e.g., write the 
letter of the correct answer, circle the cor- 
rect answe r ) . 

Develop the Items for the test . At this point 
the i terns are constructed In a ccordance with 
the test specifications. Each item should as- 
sess some aspect of the object ive(s) being 
measured. Sometimes it is possible to obtain 
previously developed items from item banks 
which have been established for just this pur- 
pose. (A brief discussion of item banks is 
included later in this section.) 

Check the appropriateness of the items . The 
content of the test items s hou 1 d be rev i ewed 
by other instructors who are familiar with the 
subject in order to determine whether they are 
appropriate in content, vocabulary and format. 
It should also be determined whether each ob- 
jective has been adequately tested by includ- 
ing enough items. The it should also be 
tried out with a small san-vr jf students to 
determine whether there a-e any problems in 
interpretation. On the basis of results of 
instructor and student reviews, the items 
should be re vised as necessary. 

Assemble the Items into a test . Decide on the 
test layout and put together the actual test. 
Ensure that appropriate directions have been 
included for each section of the test, prepare 
any necessary answer sheets, and develop a 
scoring key . 

Establish the star dards for interpreting the 
test resu Its . Deters, me, in advance, what will 
be expected of each student in order to be 
classified as having reached mastery. If the 
test includes more than one objective, decide 
(for each objective) how many items must bo 
answered correctly in order to say that tht 
objective has been mastered. If the test cov- 
ers only one objective, it will be satisfac- 
tory to indicate a percentage or total number 
correct as the criterion for mastery on the 
tes t • 
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Item Banks and Their Use for Constructing Tests 



Simply stated, item banks are collections of test items 
which have been developed to assess the mastery of specified 
objectives. These item banks may be developed and maintained 
by commercial publishers, in which case the items will typ- 
ically be sold to interested users. The item banks may also 
be maintained by a non-profit group, in which case the items 
are available for a minimal fee or on a trade basis; donat- 
ing items to the bank allows withdrawal of others. In any 
item bank, the iUms are usually grouped by subject-matter 
and the specific objective being tested. 

In general the idea of using an item bank to develop a test 
is a sound one. If the number of items written yearly by 
instructors to assess student achievement were to be count- 
ed, the total would probably include thousands of items. If 
possible, rather than writing new items each time a test is 
constructed, it would be a better use of time to go to a 
bank to select items which have already been developed; 
hence, the introduction of the item bank. 

To construct a test using an item bank, the instructor fol- 
lows the same general steps as described in the section on 
criterion-referenced test development. First, the instructor 
must identify the objectives to be measured. Then the test 
specifications are developed. However, instead of next de- 
veloping the actual items, it is here that the instructor 
make* use of the item bank. With an item bank, the instruc- 
tor chooses items that have already been written and that 
match the test specifications. This should save considerable 
time and effort on the part of the instructor. Once the 
items have been selected, the appropriateness of the items 
should be determined, the items assembled into the actual 
test, and the standards for interpreting the test results 
establ i shed. 



Deciding Whether to Use an Item Bank . In considering whether 
to use i tern banks for 1 oca I test development purposes, one 
point should be in mind: the items will vary in quality from 
item bank to* item bank. Not all item banks screen items to 
eliminate those of poor quality. In fact, some item banks 
accept any items without screening or editing. So, while the 
use of item banks can save much in the way of time and ef- 
fort and often results in high quality tests, there are cer- 
tain questions that should be asked before using a particu- 
lar bank. 



so 
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Appendix 4 - A includes a general list of questions that 
should be asked in determining whether existing item banks 
would be useful. One item bank may no* meet all the require- 
ments. As with any standardized test or set of instructional 
materials designed for general use, the instructor will need 
to identify the item bank that best suits the present pro- 
gram needs. For a brief description of sources of item banks 
see Section 5. 



ALTERNATIVE TECHNIQUES 



Although tests are useful tools for evaluating N or 0 pro- 
grams, there are often occasions when test data alone will 
not be sufficient to answer the evaluation questions of in- 
terest. This will be especially true if the evaluation ac- 
tivities focus on the understanding of a process, attitudes 
of those involved, or oth*r types of behaviors that do not 
lend themselves to ksing measured through achievement type 
tests • 

Consider, for example, the evaluation question: "Does the 
program contribute significantly to improvement in reading 
skills?" This question could be answered through some type 
of testing activity. However, if the question were "Does the 
program contribute significantly to improvement in reading 
skills and positive attitudes toward reading?", test data 
alone would not be sufficient. To collect information relat- 
ing to the question of whether student attitudes toward 
reading have changed, other types of information need to be 
collected. This might include a systematic examination of 
library records to determine whether more books have been 
checked out by the students in the last month, or a series 
of interviews with the students in the program. 

So, often it will be necessary to collect something otn 
than test information in order to fully answer the evalua- 
tion questions of interest. This section will briefly dis- 
cuss four techniques which can be used to collect other 
evaluation i n forma ti on , including: observations, interviews, 
questionnaires and existing records. The guide to selecting 
the most appropriate technique will be the evaluation ques- 
tions being asked. Additionally, because of its great rele- 
vance to N or D programs, the concept of time-on-task is 
discussed as a special application of observations. 
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Observations 



The observation 1s a method for collecting information by 
systematically watching what is occurring at certain times. 
The patterns of behavior being observed may range from the 
very simple (such as recording whether the student is in the 
appropriate place) to the very complex (such as classifying 
exactly how two students are Interacting). The person doing 
the observations may be the instructor, a volunteer aide, a 
parent, an evaluator, or even another student. In the con- 
text of the N or 0 program, the use of observations helps 
focus the data collection activities on areas not so earn^y 
measured by tests areas such as: student enthusiasm to- 
ward certain instructional approaches, the quality of inter- 
actions between the students and the instructor, and, the 
amount of time spent on instructional tasks. The use of ob- 
servational data can contribute toward a better understand- 
ing of why or how something happens and can also document 
that the event did occur. 

When considering whether to use observations to collect 
evaluation information, the following four points should be 
kept in mind: 



t Observations provide a means of collecting in- 
formation that would not be available through 
other techniques (e.g., the number of times 
positive verbal reinforcement is used with stu- 
dents ) • 

t Observations provide a means of collecting in- 
formation which does not rely on recall of what 
might have happened in the past, reducing the 
chance that events may be forgotten, over- 
looked, or distorted over time. 

t Observations provide a means of collecting in- 
formation that removes individual points of 
view from the data (e.g. s having an observer 
record and classify the types of instructor- 
student interactions rather than asking the 
instructor about the types of interactions pro- 
vides more objective data). 

t Observations provide a means of collecting in- 
formation 1n a variety of settings and with 
many types of individuals, where other data 
collection techniques may not be appropriate 
(e.g., while it might K e difficult to success- 
fully interview a group of students in a work 
•setting, an observer could document their be- 
havior through observations). 
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While there are clearly advantages to using observations to 
collect information, there are also some constraints in- 
volved in this approach. When considering whether to use 
observations to collect data, the following points should be 
cons i de red : 



• The actual presence of an observer may alter 
the behavior of those being watched. 

• The observer may interpret behavior in a way 
that is different from those actually involved 
in the dec i v i ty . 

• The observer needs to be trained in how to ob- 
serve and record behavior; the more complex the 
observation system being used, the more time- • 
consuming this training may be. 

• When fairly comp.lex behavior patterns are being 
observed, the reliability of observers can be 
an issue in interpreting results. 

• When the presence of the observer is required 
for long time periods, this technique may be an 
expensive way to collect results. 

© Because observations require a significant t 
commitment, the sample size used may have t j 
smaller than that used with other technique 



If the evaluation questions of con*, rn seem best answered by 
watching for certain events, observations should be used to 
collect the information. Appendix 4-B includes some very 
brief guidelines for developing and conducting observations. 



Time-on-Task 

One majcr instructional factor relating to achievement is 
the amount of time a student spends actually engaged in 
tasks which further his or her skills. Time-on-task , then, 
is the time devoted to tasks directly related to the devel- 
opment of the desired skills. Student performance can be 
improved by increasing the time sp<*nt on actively learning 
and practicing a skill. In general, as reflected in higher 
test scores, students learn more when they spend more time 
engaged in learning activities. 

As wouuld be expected, the amount of time students spend in 
.earning differs dramatically from classroom to classroom. 
While a student may be scheduled to attend a class for a 
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certain period of time, a variety of activities other than 
learning may occur during t .at time. For example, students 
may b? engaged in socializing, obtaining materials with 
which to work, recordkeeping, being disciplined, or other 
non-i nstructiona 1 activities. Of course all class time can 
not be used directly in learning skills, but a good portion 
of time should go toward skill acquisition. 

When evaluating a program 1t may be necessary to determine 
how time is actually being used in the classroom. If, for 
example, the evaluation question is: "What portion of in- 
structional time is directly spent on tasks related to ob- 
jectives of the program?" or "How effective are the instruc- 
tors in managing classroom activities? 1 ^ then it would be 
necessary to determine how time is actually used. Likewise, 
if the results. of past evaluation activities have indicated 
that students are not improving their skills, the use of 
class time may become an issue. In any of these situations, 
1n order to answer the questions being asked, it will be 
necessary to systematically observe the classrooms in order 
to document what is occurring. Clearly the purpose in deter- 
mining how time is spent in the classroom is to increase the 
amount of time-on-task , thus increasing student learning. 
To accomplish this it is necessary first to determine exact- 
ly how time is be^ng used and then to red* ce the 
non-Instructional user where possible. So, whsn ooserving 
the classroom it will be necessary to document how much time 
is devoted to learning end how much time is used for other 
activities. 

Engaged time can be used on intoractive activities in which 
the student is working on instructional tasks with others 
(e.g., the instructor, an aid*, or other students) or on 
non-1nteracti ve tasks in which the student is working alone. 
Engaged time, then, includes activities such as: 

competing in drill and practice games, 
participating in a discussion, 
taking part in role-playing activities, 
listening to a lecture, 
asking questions, 
participating in a demonstration, 
receiving feedback on some work, . 
reading in i nstructiona! manual, 
working on written assignments, 
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• watching a filmstrip, or 

t working with a microcomputer program. 



Non-engaged time, on the other hand, includes activities 
such as: 



• working ors other assignments, 
t socializing with others, 

• obtaining work materials, 

• ansveri ng to roll call, 

• filling out health forms, 

• being i i s c i p 1 1 n e d , 

• recordi ng progress, 

• pas -t - . Jig t papers , 

t being calleo out of the room, 

• observing others, or 
a doing nothing. 



While some of these activities, such as o b t a *i n i ; g work 
materials, are necessary to the task of learning, others 
a^e not a very effective use of class time and should be 
reduced whenever possible. 

After the time-on-task observations have been completed it 
will be possible to look at the total picture of student ac- 
tivities to determine how to increase the effective use of 
cUss time. How much time is spent on instruction? On non- 
instructional tasks? Do students take too long to get ready 
to work? Are too many administrative activities reducing the 
time available for i nstructi ona 1 purposes? Understanding how 
time is really used will indicate where changes cou ] d be made 
in order to increase ths amount of time students spend en- 
gaged in instruction and, ultimately, will increase student 
1 ea rn i ng c 



Ques ti onnai res 

The questionnaire presents individuals with a series of 
carefully developed questions covering a predefined topic 
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and requires some manner of written response. Question- 
naires may include items that are open-ended (requiring the 
respondent to write in some form of response), closed (re- 
quiring only that the respondent select an answer from 
choices provided), or both* 

For N or D programs, the use of questionnaires allows the 
collection of information through methods such as: surveying 
those who employ graduates of the program in order to deter- 
mine employer satisfaction and to pinpoint areas in which 
changes might h> roace to improve the program; surveying past 
students to dettv ?r .ine whether the program was effective for 
them; determining community attitudes toward the program; 
and documenting parental attitudes toward the program. In 
general, the use of questionnaires facilitates the collec- 
tion of a wide variety of information from a large group of 
i ndi v i dua 1 s . 

In considering whether to use questionnaires to collect 
evaluation information, the following five points are of im- 
portance : 



t Questionnaires provide an inexpensive means of 
simultaneously collecting information f r r,i a 
large nunioer of people. 

• Quest ionnai res provide a means ensuring the 
respondents' anonymity, which sometimes results 
in more honest responses to sensitive ques- 
tions . 

t Questionnaires provide a means of asking uni- 
form questions to everyone, thus ensuring that 
the necessary data is collected from all in- 
volved. 

• Questionnaires, especially those using closed 
item formats, provide a means of collecting 
data which is fairly easy to summarize and in- 
t e rp ret . 

t Questionnaires provide a means of collecting 
information over many topics of interest, rang- 
ing from general attitudes to details on past 
experiences. 



Clearly there are many advantages to using questionnaires, 
but there are also constraints which should be noted* in- 
cluding: 
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• If a question is unclear to the respondent, it 
cannot be clarified and as a result might go 
unanswered . 

• Those asking the questions may not be able to 
follow up on interesting lines of thought or 
probe for more detailed responses, 

• Because questionnaires are somewhat impersonal, 
the response rate may be low and those who do 
respond may constitute a biased samplp. 

• Some respondents may have difficulty in reading 
or in expressing answers in writing. 



If the evaluation questions of concern seem best answered by 
asking a number of people a series of written questions, 
questionnaires should be used to collect the information. 
Appendix 4-C includes some very brief guidelines for devel- 
oping and administering questionnaires, . 

Int e rvi ews 

The interview is a method for collecting information by ask- 
ing a series of questions of each individual included in the 
sample. R.ifter th*n requiring the respondent to read a 
question and answer it in writing, the interviewer asks each 
question, carefully recording the oral response given. The 
int ^viewer lay then systematically follow w the re- 
sponses, either through an informal ap|^<vcb -j« hrough a 
predetermined s^t of additional questions. 

For N or D programs, the interview facilitates the collec- 
tion of information from individuals who may have difficulty 
reading or writing, where non-verbal reaction? are highly 
relevant, and where detailed probing of responses is neces- 
sary. Interviews would be on appropriate means of collecting 
information on areas such as: the types of instructional 
approaches that the students prefer; the types of interper- 
sonal skills that potential employers would like future em- 
ployees to have; or the ways in which instructors would like 
to see the program reorganized. 

When considering whether to use interviews to collect eval- 
uation information, the following six points are relevant: 

• Interviews provide a means of collecting infor- 
mation which does not depend upon the reading 
or writing skills of the respondent. 
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Interviews provide a means of collecting infor- 
mation in a manner which first allows rapport 
to be established between those involved. 

Interviews provide a means of collecting infor- 
mation on non-verbal responses, language and 
voice inflection, as well as the verbal re- 
sponse. 

Interviews provide a means of collecting de- 
tailed information through the use of rephras- 
ing of questions and further probing of re- 
sponses . 

Interviews provide a means of asking questions 
that may be difficult to phray", in writing or 
which require fairly extensive clarification. 

Interviews provide a nieans of collecting all of 
the information from rhose who participate, 
avoiding the possibility that responses may be 
missed due to unclear questions. 



Interviews, of course, have constraints which should be con- 
sidered before deciding to use the techniques. Some of the 
constraints are as follows: 



• Because the results of the interview rely 
strongly on the interviewer's interpersonal 
skills and communication capabilities, some 
respondents may be threatened, led toward cer- 
tain responses, or be generally uncommunica- 
tive. 



The interviewer, unless carefully trained, may 
get off track, alter the meaning of questions 
by slightly rephrasing a few words, fail to 
follow \ip on responses where appropriate, or 
miss key non-verbal nuances. 



t Because this technique requires the presence of 
an interviewer at all times, the interview can 
be an expensive and time-consuming way to col- 
lect information; smaller sample sizes may be 
necess i tated . 

t Without careful planning, interview data can be 
difficult to summarize and interpret. 



If the evaluation questions being asked seem best answered 
through the oral administration of a set of questions and 
further probin, v.,, responses, then interviews should be used 
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to collect the information. Appendix 4-0 includes s,?r-, v> 
brief guidelines for developing and conducting i r. ; e ■ «sw 



Existing Records 

Existing records, although not an information collection 
technique, are clearly a source of evaluation information. 
The use of existing records is a method of collecting infor- 
mation on the basis of what has already been compiled in 
some manner by others. This would include any kind of data 
which has been systematically collected at a previous date 
or obtained as a byproduct of other activities. 

Existing records encompass a broad range of information, in- 
cluding: records maintained for student management (e.g., 
plass or individual progress charts, individual student 
files, student scores on progress tests); records maintained 
for short- and long-range planning purposes (e.g., objec- 
tives taught in the program, services available outsic-e of 
the program, staff backgrounds); and records maintained for 
evaluation and administrative reporting purposes (e.g., 
att en dance, discipline reports, standardized achievement 
tests, funds used for spec i a 1 i zed . eq u i pme n t ) . 

In the N or D program, existing records can provide a source 
of information to answer many types of evaluation questions. 
(Section 6 deals in detail with the types of records which 
should s e maintained and their use for program evaluation 
and management.) Further, existing records can provide the 
background information necessary to complement data collec- 
ted in other ways. For example, while interviews may be used 
to collect information on students' attitudes toward certain 
instructional materials, existing class records can docume.- 
how well the students have learned by using the various ma- 
terials. Combined, the two pieces of information provide a 
more complete picture for evaluation purposes. 

When considering whether to use existing records for evalu- 
ation purposes, the following four points should be kept in 
mind: 



• Existing records provide a wide variety of 
readily available information. 

• Existing records, because they have not been 
interpreted by others, are generally • source 
of fairly objective information. 

t Existing records are generally considered a 
credible source of information because the data 
has been collected at the time of the event, 
rather than recalled at a later date. 
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• Existing records provide information which is 
low in cost to collect and may be obtained in a 
sorter time period :han that required to get 
new i n format i on . 



There are, of course, constraints which should be kept in 
m.;d when considering the use of existing records, includ- 
ing: 



• Existing records may be incomplete, with gener- 
ally no way tu retrieve this missing informa- 
tion. 

• It may take some time and effort to extract the 
desired information from the existing records 
(e.g., the last five years of the test scores 
rcay be available, but stored in a box in the 
basement of another building). 

• Permission to use existing records may involve 
some legal requirements, such as permission 
from the individuals whose records are of 
i nterest . 



For ill evaluation questions being asked, consider the 
fear^h i;ty of using existing records either to answer the 
que& !;<..; or to provide supplementary information necessary 
to f i i ; aut the picture. Appendix 4-E includes some very 
brief guidelines for the use of exiting records. 
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Appendix 4- A 
QUESTIONS TO ASK REGARDING AN ITEM BANK 



1. What kind of information is available about the indi- 
vidual items in the bank? 



The information needed about items will differ iipy^ 
on the academic area. Constructing math tests fr -° 
banks, for ..stance, does not require as much a 
tion about items as construct ing reading tests. 

a. Can the appropriate grade level be identi- 
fied for which an item is appropriate? 

This is usually not difficult for math 
tests; .nath items are usually described by 
a particular operation that is taught at a 
certain grade level. For reading items, 
however, if the instructor is looking for 
an item where the student must identify 
the main idea, he or she will probably 
want a way of knowing the reading level of 
a passage without having to actually pull 
the item from the bank first. 

b. What kind of information is available 
about the techn^' ca 1 . qua 1 i ty of the items? 

Can the instructor tell how difficult the 
item is for different grade levels of stu- 
dents? Most instructors prefer that the 
test contain both easy and difficult it ems 
to allow students to show what the*' do 
know and to find what they do not know. 
Also s a check must be made to see if this 
information about the item has been up- 
dated. 

c. Is there a way that instructors can use 
the students' incorrect answers to diag- 
nose thei r problems? 

Often instructors like to use the results 
from a test in the diagnosis of their stu- 
dents 1 strengths and weaknesses. It helps 
If the instructor can identify a problem 
by using information from a wrong answer 
that was chosen in the test. 
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Appendix 4-A (Continued) 



d. How specific can the instructor be when re- 
questing i t s. -lis ? 

For example, suppose the instructor would like 
an item ' ^asure the recognition of the 
consonant blend "CL". Is it possible to pull 
items directly from the bank that deal with 
"CL" or must all items dealing with consonant 
blends be searched through to locate i terns 
dealing with "CL"? 

What kinds of quality control measures have been applied 
to the items entered into the bank? 

Have the items been reviewed by instructors and curricu- 
lum experts for correctness of the answers? Is there a 
guarantee that the items really do measure the skills 
they profess to measure? Have the items been reviewed 
for possible biases such as toward different sex, 
ethnic, racial or regional groups? 

What kinds of item response formats are available? 

Does the bank include items in a variety of response 
formats and is t!;ere an option when choosing the items? 

What :re the ■ ual procedures that must be followed 
when "j'ng the *em bank? 

Some developers of .item banks request the requirements 
for a cost and will deliver either the options for the 
items or the actual test. Others will supply the actual 
item bank. 

A good suggestion here is that when an item bank is 
considered, a test run should be made involving the 
actual- persons who will be using it. RecoMs should be 
kept of what has to be done, how long it takes, how 
difficult it i c and how it compares to what has been 
done in the past. 

How does the organization of the item bank match the 
instructor's curricular orqan i za t i on ? 

How difficult will it be to locate the sections of items 
in the bank that deal with particular sections of 
instruction? In some cases, the instructor will find 
that it is easier to adopt the objective system of the 
item bank than to translate program objectives to the 
objectives of the item bank. 
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6. What is the cost? 

Developing an item bank can be a very costly venture. 
Buying an item bank or contracting with an item bank 
service can also be expensive, A careful analysis 
should be made to determine whether the advantages of 
the item bank outweigh these costs. 
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Appendix 4-B 

GUIDELINES FOR DEVELOPING AND CONDUCTING OBSERVATIONS 



A. WHEN PLANNING THE OBSERVATION: 

1. Identify the category of behaviors on which 
data will be collected. Limit the category to 
one small enough to be reasonably done durirrg 
an observation session. Do not expect to col- 
lect information on every behavior of intrust 
at one t i me • 

2. Determine who will be observed. The sample will 
affect how the observations are done, the 
length of the observation, and the system for 
doing the actual observation. 

3. Decide ahead of time how the results of the ob- 
servations will be analyzed. The data analysis 
can affect the format of the observer recording 
sheets and the types of information actually 
col lected. 

4. Limit ooservations to areas in which informa- 
tion cannot be collected in other ways. For 
example, usinp observations to obtain informa- 
tion on the age of some students would not be 
the best method of data collection. On the 
other hand, direct observation to determine 
eye-hand coordination of students would be ap- 
propriate . 



B. WHEN DEVELOPING THE OBSERVATION INSTRUMENT: 

1. Identify and clearly define each behavior that 
the observers will be looking for. The explan- 
ation of a behavior should not be vague, gene* - 
a 1 , or open to interpretation by the observerSr 
Having observers watch for disruptive behavior 
would result in very unreliable data. Exactly 
what is disruptive behavior? In comparison, 
having observers tally the number of times a 
student left the seat would be a behavior much 
less open to interpretation, result i no in more 
re 1 i a b 1 e dat a . 
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Appendix 4-B (Continued) 



2. Develop a coding method, tally sheet, or other 
device that facilitates the observation pro- 
cess. If the observer has to take time out to 
write down words, the behaviors occurring dur- 
ing that t i me will be lost. 



WHEN PLANNING TO USE THE OBSERVATION: 

1. Ensure that each observer is fully trained, in 
th« procedure. This would include an under- 
standing of the definitions of each behavior, 
practice at using the device on which the data 
will be recorded, and how to be unobtrusive 
while doing an observation. If the purpose of 
the observation is simply to describe events as 
'„ !ey occur rather than watch for specific be- 
haviors, the observer still needs to be trained 
in methods for recording behaviors. 

2 . When scheduling the observations, keep each 
period fairly short. 'Observing and recording 
behaviors is a very intense activity, so should 
be divided into several brief periods, rather 
than one long one. For example, if the observer 
needs to watch a classroom for a total of 30 
minutes, ten three-minute series of observa- 
tions would provide better data than three ten- 
minute periods. Of course, there may be times 
when the purpose of the observation is to de- 
scribe what went on during an entire lesson, in 
wh'ch case it would not be possible to break up 
the observation periods. 
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Appendi x 4-C 

GUIDELINES FOR DEVELOPING AND ADMINISTERING QUESTIONNAIRES 



A. WHEN PLANNING THE QUESTIONNAIRE: 

1. Identify the topic of the quest ionnai re. Decide 
on this topic before beginning to develop the 
questions and stay within that area. 

2. Determine the intended audience before the 
questions are developed. Audience characteris- 
tics will affect the format of the entire ques- 
tionnaire and the phrasing of each question. 

3. Determine ahead of tir.*e how the questionnaire 
results will be analyzed. This will affect the 
format of the included questions. For example, 
if the questionnaire responses are going to be 
machine-scored, open-ended questions could not 
be used. Or the closed format might be used and 
respondents asked to answer on a separate sheet 
which could then be scored directly by the com- 
p u t e r . The questionnaire should al r o be ar- 
ranged to facilitate scoring responses by 
grouping similar items together (i.e., all 
yes/no type questions together). 

4. Ask only for information which cannot be ob- 
tained elsewhere. The purpose of the question- 
naire is to collect some type of information or 
attitudes from each individual. Each person is 
responding because his or her i^put is neces- 
sary and of interest. If the information can be 
obtained elsewhere there is no reason to have a 
person spend time repeat ing that information. 

5. Keep the questionnaire short. A person is much 
less likely to respond to a long questionnaire 
and more likely to return the questionnaire if 
it is of reasonable length. 

B. WHEN DEVELOPING THE QUESTIO/5 TO INCLUDE ON THE QUES- 
TIONNAIRE: 

Order the questions in a logical manner. Start 
with the most general types of questions, then 
move on to the specifics. 
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Appendix 4-C (Continued) 



2. Limit each question to one idea. Do not combine 
more than one idea in a single question. If a 
question does cover more than one point it is 
impossible to interpret the results later. 

3. Do not ask leading questions. Be sure that the 
questions do not lead the respondent toward the 
desired response. If the respondent can tell 
what the "correct" response is, then the ques- 
tionshouldberewritten. 

4. Word each question as simply and clearly as 
possible. Do not include information that i 
unnecessary to the question and avoid technical 
terms, unless they are appropriate to the audi- 
ence. 

5. Include a definite point of reference to ensure 
that each individual responds to the same ques- 
tion. For example, if a question is asked, "How 
many hours do you work?," respondents may an- 
swer in terms of hours per day, per week, or in 
other ways. Changing the question to ask "How 
many hours per day do you work?" ensures that 
each individual responds to the question in the 
same manner. 

6. If a closed question format is used, try to in- 
clude options that cover all possible aspects 
of that question. Do not limit the answers to 
only one side or part of an issue. Addition- 
ally, since it is often difficult to anticipate 
all possible choices to include in a closed 
format, use the category "other" and allow a 
space for the person to write in a response. 



WHEN PLANNING FOR ADMINISTRATION OF THE QUESTI ONAI RE 

1. Include complete and clear directions on how to 
respond to the questionnaire. Explain exactly 
how to respond and where. Ho not leave anything 
up to the respondent's imagination. 

2 . Include a cover letter with the questionnaire. 
This letter should be addressed to each respon- 
dent. The purposeof this letter is to estab- 
lish rapport with the respondent, to explain 
why the questionnaire is being sent, and to 
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encourage that the questionnaire be returned. 
It is a good idea to include a deadline date 
for returns in this letter. 

3. Ensure that the questionnaire and all other 
correspondence is neat and easy to read. A 
poorly arranged questionnaire or one that is 
difficult to read will have less of a chance of 
being returned than one which is we 1 1 -des i gned . 

• 

4. Include a stamped, se 1 f -a ddressed envelope with 
the questionnaire. This will help encourage re- 
turns. 

5. Use postcards or other means to fellow up on 
those questionnaires that were not returned. 
Remember that the more questionnaires returned, 
the less biased the sample. 



ERLC 
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Appendix 4-D 

GUIDELINES FOR DEVELOPING AND CONDUCTING INTERVIEWS 



WHEN PLANNING THE INTERVIEW: 

1. Identify the one topic of the interview, 

2. Determine the Intended audience, 

3. Determine how the results will be analyzed, 

4. Ask only for information which cannot be ob- 
tained e 1 sewhe re • 



WHEN DEVELOPING THE QUESTIONS TO ASK DURING THE INTER- 
VIEW: 

1. Order the questions in a logical manner, 

2 . Limit each question to one idea. 

3. Do not ask leading questions. 

4. Word each question as simply and as clearly as 
possible . 

5. Include a definite point of reference to ensure 
that individuals respond to the same questions. 



WHEN PLANNING FOR ADMINISTRATION OF THE INTERVIEW: 

1. Establish a method for recording interviewee 
responses. It is important that the actual 
wording be preserved as closely as possible. 
Recording methods include taking notes during 
or after the interview, using preplanned re- 
cording sheets, or taping the session. 

2 . Train the individuals conducting the interview. 
They should be able to conduct each interview 
in the same manner, ask the same questions in 
the same order, and avoid any emotional re- 
sponses to the Interviewee's answers which 
might affect future responses. The interviewer 
must be trained to constantly probe for ad- 
ditional Information and accurately note re- 
sponses to each question. 
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The interviewer should also be trained to 
the first few minutes of the interview to 
tablish a good rapport with the interviewee 
explaining the purpose of the interview. 
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GUIDELINES 



Appendi x 4-E 
FOR USING EXISTING RECORDS 



A. WHEN DECIDING TO USE EXISTING RECORDS: 

1. Identify the area of information for the data 
collection. Once the area is established, ap- 
propriate types of existing records can be s*e - 
1 ected . 

2. Describe the sample on which the information 
will be collected as clearly and completely as 
possible (for example, fifth grade students who 
have attended elementary schools in the dis- 
trict for the last two years). 

3. Decide exactly what type(s) of records will be 
used. Considering feasibility, cost, access, 
time, and legal implications will help in de- 
termining this. 

4. Determine ahead of time how the information 
will be analyzed. The process used for data 
analysis will have implications for how the 
information is recorded. 



B. WHEN PLANNING TO COLLECT INFORMATION FROM EXISTING 
RECORDS: 

1. Identify exactly where the necessary informa- 
tion is located and whose permission must be 
obtained in order to access these records. 

2 . Determine ahead of time any legal requirements 
which must be met in collecting or using the 
records. 

3. Develop a method for extracting the necessary 
i nf ormat 1 on ,and a means of recording that data. 
The method used must be easily understandable 
and cons i stent . 

4. Train those who will be collecting the informa- 
tion. To obtain reliable data, each individual 
must record the same type of information. 
Therefore, each person must clearly understand 
any categories on the recording forms and di- 
rections for their use. 
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5. INSTRUMENTATION 



INTRODUCTION 



Selecting the appropriate type of instrumentation is a very 
important part of the evaluation process. Section 4 dis- 
cussed the application of testing and other types of infor- 
mation collection techniques used to answer evaluation 
questions. This section provides specific Information on 
criterion- and norm-referenced tests, measures of affective 
behavior and sources of item banks. Selected references for 
further information on instrumentation are also provided. 

AN ANNOTATED BIBLIOGRAPHY 

The bibliography which follows presents information about 
the various tests and other measurement techniques available 
for evaluation of N or D programs. 

The annotations provided in this bibliography we re 
prepared only to serve as an information resource 
and are not intended to imp'y endorsement or ap- 
proval for use in Chapter 1 evaluations, 

Clearly there are many other tests and measurement tech- 
niques which could have been discussed in this section; keep 
in mind that the ones discussed here are only examples of 
the potential choices available for evaluation purposes. 

Prior to making a final determination regarding the choice 
of instrumentation, the actual instruments and related pub- 
lications should be carefully examined. Review copies or 
specimen sets of most instruments are typically available 
from the pub 1 i shers f o r just this purpose. The appendices 
which follow this section will also be of help in making a 
final selection. 



Appendix 5-A and 5-B, respectively, provide rating scales 
for selecting criterion- and n o rm- re f e re n ce d tests. Each 
rating scale provides a series of questions which, when 
asked in relation to a specific test, wil 1 help determine 
whether that test is an appropriate choice for the evalua- 
tion activities. When using these rating scales it should 
be kept in mind that no one test will be perfect for the 
program evaluation activities, but some tests will be better 
than others . 
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Appendix 5-C provides a process by which a test review team 
can look more closely at the items in a test in order to 
determine how well the test measures the program's objec- 
tives,, Upon completion of this process the user will be able 
to compare this information across tests to help to deter- 
mine which test best matches the program objectives. 

Appendix 5-D provides some guidelines for determining when 
to test out-of-level . There will be occasions when the per- 
son planning the test administration feels that the pub- 
lisher's recommended test level may not be appropriate for 
the student(s) taking the test. If there is some question as 
to whether the test level will be too easy or too difficult, 
out-of-level testing should be considered. 

Finally, Appendix 5-E provides a test administration check- 
list which can be used to ensure that the actual testing 
goes as smoothly as possible and is done correctly. Follow- 
ing incorrect test administration procedures can result in 
test data which may not be an accurate reflection of the 
student's scores. Therefore, following the appropriate test- 
ing procedure is very important to the evaluation process. 

List of Annotations 

Adult Basic Learning Examination, 1967-74 
Attitude Toward School, Rev. Ed., 1972 
California Achievement Tests, 1977-78 

Comprehensive Assessment Program Achievement Series, 1980 

Comprehensive Tests of Basic Skills, 1981-82 

DIAGNOSIS: Mathematics Level B 

Gates-MacGini tie Reading Tests, 1978 

Instructional Objectives Exchange 

Iowa Tests of Basic Skills, 1978 

Mathematics In Our World, Second Edition, 1981 

Measures of Self Concept, Rev. Ed., 1972 

Metropolitan Achievement Tests, Instructional Battery, 
1978-79 

Metropolitan Achievement Tests, Survey Battery, 1978-79 
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Northwest Evaluation Association Item Banks 
PRISM 

Reading Yardsticks, 1982 

Sequential Tests of Educational Progress, Series III, 1979 

SRA Achievement Series, 1978 

Stanford Achievement Test, 1982 

Stanford Diagnostic Reading Test, 1973-78 

Stanford Diagnostic Mathematics Test, 1976-78 

Tests of Adult Basic Education, 1976 

Wide Range Achievement Test, 1978 
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ADULT BASIC LEARNING EXAMINATION Achievement: Adult 

(A3LZ), 1967-74 (Nona-Referenced) 



ABLE is a battery of testa developed to measure the level «* a ,u; a „ 

literacy 1 0 w associated with basic education or functional 
varilrv If . A J^ OU8h Ce3C concenC " adult-oriented, A3LE nay be used in a 
Ali:Z^\i:il^ a C ° -hiev«enc from the primary gLdes" cne* 

SL^I" K eVel3 ° f Att ?' 6aCh in alternate Forms A and B 

«2e L e virr enC /^ iCa i ° f 8rade P"*""™" from first through' twelfth 
K^Ili ri 1 13 de3l8ned for achievement levels in grades 1-4- Level II 
in grades 5-8; and Level III; i n grade3 9 - 12 . S * L6Vel 

Each level of ABLE consists of four tests: Vocabu- larv (which reauires 

aspects of practical life, such as community, family, and job AST* 

o a ^divL aCl r r K 3UU3 " n " Veal Che "«F«*<ive strength a„d Tessas 
o. individuals; however, they are not intended to provide indeoch dia^as- 
cic inxormacion for instructional purposes. P"viae i^epcn diagnos- 

Amr C V C l K°\f aCh leVeI ind f0ra are P ub l"hed in a separate booklet The 

I booklets, smaller in size than conventional test booklets Ire 
color coded and include practice items. The number of test items Ear ILh 
test or subtest vary by level- for L«v-i t pk. • ce3C . lCea3 for each 
F .„ - . 7 , *. ' ror Level Ii the numoer of items ranzes 

rllV.TJ L T l l lt fr ° B 20-585 and for Level Hit from 42-60 The 

response mode for Levels I and II is varied; all items in Level III' .r* 

hand?"? ^^'^ £ °™ C ' ^ 1 « d 11 » «S ^ e in both 
hand- or machme-scorable test booklet editions. Level III requires the use 
of a separate answer sheet which may be scored by hand or machine 

The entire battery of tests for each level can be administered in a DD rc*i- 
mate y two hours. However, each of che subtests may be administered in a 
single session. Administration time for Level I ar Z iz a ° m J ni3Cered ) a 3 
20-25 minutes and for Level III, < T Jll Tel minute ' 'mo ITlVJlTe 

:c^u l Ied S1 oe e tve^ a t:scs 8 ! e ""^ * ~ B ^ ° f * 

whi^'tf^l' VXS • 5 " iCe, ° 3C f eenin * Ce3C " available to help decarmine 
which level of ABLE is most suitable for use with an individual. The Jes - 

?5 min b ° Ch . Verbal and «u«.ric«l concepts, is untimed buc takes 111*1 

or subtLr C ° adain i 3 "f- screening- test and an addicional ABLE test 

or subtest can be administered in che same testing session. Directions for 
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ADULT 3 ASIC LSA-tNING EXAMINATION (continued) 



test administration are provided in che A3 LI Handbook that accomoanies each 
test level. An ABLE Group Record form (one for each level) is available fox 
recording students' scores. 

In Levels I and II, che number of item* correct for each cesc or subcesc is 
converted to a grade score. The grade norms vere escablished in 1966 by 
equating ABLE wich che Stanford Achievement Tesc, 1964 edition. The grade 
norma are based on che performance of a sample of approximacely 1,000 stu- 
denes per grade in grades 2-7, drawn from four school systems in four 
states. The grade norms provide a rough indicacion of individual pferforaanc 
andsuggest che level of instructional materials Co be used. Split-half 
reliabilicy coef ficiencs , correcced by che Spearman-Brown formula, for four 
research groups (scudencs from grades 3 and 4, Job Corps enrollees, and 
adult basic educacion students in Hartford-New Haven, ranged from .73 to 
.98 for Level I and from .60 to .96 for Level II. 

Percentiles and stanines for Level III were obtained in 1970 by equating 
ABLE with the Stanford Achievement Test: High School Battery, 1965-66. 
Reliability coefficients (K3-21), obtained from two school groups and five 
adult groups, ranged from .81 to .96. 

ABLE is available from The Psychological Corporation, 757 Third Avenue, New 
York, New York 10017, (Phone: 212-888-3500) or from the publisher's 
regional offices. The 1982 catalog. price listed for a specimen set is 
$5.25 for each level; and a package of 35 test booklets is S32.50 for Level 
I, $33.75 for Level II, and $34.50 for Level III. * 
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ATTITUDE TOWARD SCHOOL, REV. ED., 1972 Affective: X-22 

This L83-page collection of measurable objectives and related assessment 
instruments prepared by the Instructional Objectives Exchange ClOX), is 
devoted entirely to attitude toward school. The collection published in 
paperback book format, contains complete tests, along with description and 
rationale, directions for administration, and scoring guides. 

The affective measures are intended for vise in pretest/postttest evaluations 
of programs designed to improve student attitude toward school. From among 
42 clearly defined objectives and related measures, users may select those 
which they consider to be appropriate for their instructional settings. 
Local modifications to the measures may be made if particular items are 
considered inappropriate. Items may be deleted, modified, or added. 
However, care must be taken that changes are consistent with ehe objective 
to be measured. The measures are designed to be used for assessment of 
group attitude only and not for individual assessment. 

The attitude toward school objectives and assessment measures are arranged 

r? C ?,? hre t. grade UvelS: Primar 7 ( *- 3 >. Intermediate (4-6), and Secondary 
17-12). The measures focus on five dimensions of attitude toward school: 
teachers, school subjects, learning, school social structure and climate, 
peers, and general orientation toward schooling independent of a particular 
school. The measures include three types: direct self report instruments 
which solicit student reaction in a direct question-answer format; infer- 
ential self report measures which permit inferences based on indirect 
stimuli questions; and observational indicators which permit inferences 
based on direct observation of student behavior. An overview of the 
ueasures included in the collection follows. 

Direct Self Report Measures 

School -Sentiment Indax (Primarv/Intermediate/Secondary ) 

SSI, an omnibus invencory available in a separate version for each test 
level, assesses five dimensions of attitude Coward school: teacher 
school subjects, school social structure and climate, peer, and general. 
Students exhibit favorable attitudes by indicating agreement with 
statements that reflect positive perceptions, *nd disagreement with 
statements that reflect negative aspects of ehe various dimensions. 
The SSI includes 37 items for the Primary level, 81 items for the 
Intermediate level, and 82 items for the Secondary level. Administra- 
tion time is approximately 10-15 minutes, 20-30 minutes, and 15-20 
minutes for Che Primary, Intermediate, and Secondary levels, respec- 
tively. It is recommended chac che SSI be adminiscered by someone ocher 
Chan che ceacher co minimize a bias effecc on the studsncs* responses. 
Scores available include a single global score and subscale scores for 
posicive acticude Coward school. Reliabilicy coefficiencs for SSI 
cocal scores are .87 (cesc/recesc) and .72 (KR-20) for che Primary level; 
.83 (cesc/recesc) and .80 (KR-20) for che Inceraediaee level; and .49 
(cesc/recesc) and .88 (KR-20) for che Secondary level. Tesc/recesc 
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ATTITUDE TOWARD SCHOOL (continued) 



coefficients for the SSI subscales across all levels ranged from 
.35 to 90; KR-20 coefficients for the subscales across all levels 
ranged from .42 to .70. 

A Picture Choice (Primary, Grades K-l) 

This assessment measure focuses on a student's interest in several 
subject areas (language, listening and speaking, science, and 
aesthetics (art and arusic) . The instrument presents 28 sets of 
three hypothetical activities from which the child selects those 
he/she would like most to do. It is assumed that relative interest 
in the various subject areas may be inferred from the activities 
which the child selects. The activities are presented both orally 
by the test administrator and visually in the form of a picture on 
the student's response sheet. Scores are obtained for each subject 
area. Administration time is about 20 minutes. Reliability data 
is not available. 

A Picture Choice (Primary, Grades 2-3) 

This instrument requests students select one activity that he/she 
would most like to do from each of 30 sets of hypothetical activities 
in different subject areas. Each activity is presented boch orally 
by the test administrator and visually in the form of a picture on 
the student's response sheet. Administration time is approximately 
20 minutes. Reliability data is not available. 

Inferential Self Report Measures 

Subject Area Preferences. (Intermediate/Secondary) 

This instrument is composed of a list of subject areas commonly 
taught in junior and senior high school. Each subject area is 
accompanied by seven-degree scale on which students mark their 
relative preferences. This inventory provides an index of students' 
relative preferences among the given subject areas. Reliability 
coefficients for each subject area subscale across both levels 
range from .53 to .86 (test/retest) and .45 to .74 (KR-20). 

Imagine That (Intermediate/Secondary) 

This inventory presents 10 or 11 (for the Intermediate or Secondary 
levels, respectively) hypothetical situations regarding teacher 
behavior in the following areas: mode of instruction, authority and 
control, and interpersonal relationships with pupils. The student 
selects one of four alternatives for each situation. A score is 
obtained by totaling the number of positive alternatives selected. 
Reliability coefficients for the Intermediate level are .79 (test/ 
retest) and .62 (KR-20), and .51 (test/retest) and .58 (KR-20) for 
the Secondary level. 

Region V TAG 
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ATTITUDE TOWARD SCHOOL (continued) 



The Story (Intermediate) 

This instrument asks the student to select statements from a list 
that would fit in a realistic story about the student's school. 
The items describe situations expressing perceptions of the student 
and his peer group. The instrument is based on the assumption that 
a student's perception of the peer group is a component of a general 
attitude toward school. Reliability coefficients are .75 (test/ 
retest) and .68 (KR-20). 

Looking Back (Intermediate) 

This instrument consists of 14 statements concerning remembrance 
of positive feelings about school that a person might make when 
thinking back about school years. The total score consists of the 
number of positive responses made by the student. Reliability 
coefficients are .86 (test/retest) and .67 (KR-20). 

The School Play (Intermediate) 

This instrument consists of 19 sentences that state positive and 
negative perceptions of the structure and general climate of a 
school. The students are asked to select those statements that 
could be used to write a play about their school. The score 
obtained is the total number of positive statements selected. 
Reliability coefficients are .69 (test/retest) and .74 (KR-20). 

What Would Happen (Secondary) 

This instrument consists of 11 fictitious situations involving two 
new students at school. Students are instructed to pretend that 
they are writing a short story and to select from among four alter- 
natives to each situation which describe what would probably happen 
to the new students at their school. The reliability coefficient is 
.54 (both test/retest and KR-20). 

Take Your Pick (Secondary) 

This measure presents students with 12 hypothetical situations, each 
with four alternative responses. Student scores consist of the 
number of alternatives selected which indicate a tendency to approach 
rather than to avoid learning-related activities. A dmi nistration 
time is about 5-10 minutes. . Reliability information not available. 

High School on T.V. (Secondary) 

This measure includes 12 hypothetical situations relating to the 
school social structure and climate. The student is asked to pretend 
that he/she is writing a television script about his/her school and 
to select from among three alternatives the one which depicts the 
most realistic details, based on their own school experience. 
Students are scored according to the number of positive alternatives 
selected. Administration time is 5-10 minutes. Reliability coeffi- 
cients are .61 (test/retest) and .54 (KR-20) . 
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ATTITUDE TOWARD SCHOOL (continued) 



Observational Indicators 

Compliance With Assigned Tasks (rrimary/Intermediate) 

This assessment measure includes an observation record form which 
lists several tasks in which students might be ejected to be engaged 
in the classroom. The measure is based on the finding that com- 
pliance with assigned tasks is a correlate of general liking of 
school. Data is to be collected by an outside observer. Directions 
for administration and scoring are provided. 

• 

School Conduct: Compliance With School Rules (Primary/Intermediate) 

This observation record form utilizes available school records 
regarding pupils referred to school authorities. The assessment 
measure is based on the concept that pupils possessing favorable 
attitudes toward school will tend to accept the school rules and 
abide by them. Directions for administration and scoring are 
provided. 

School Tardiness (Primary /Intermediate) 

Records may be kept and an average daily tardiness rate computed 
a specified time period. It is assumed that students who have a 
positive attitude toward school will tend to arrive at school on 
Directions for administration and scoring are provided. 

School Attendance (Primary/Intermediate/Secondarv) 

Attendance records may be observed during specified time periods. 
It is assumed that students who possess favorable attitudes toward 
school will tend to incur a mln-firmm of absenteeism. Directions for 
administration and scoring are provided. 

Class Attendance (Secondary) 

Attendance records f.or individual classes may be observed. It is 
assumed that students who hold favorable attitudes toward specific 
c la sses or subjects will tend to incur a minimum of absenteeism from 
those classes. Directions for administration and scoring are 
provided. 

Class Tardiness (Secondary) 

Tardiness records may be utilized as an observation indicator. It 
is assumed that students who hold a favorable attitude toward their 
classes will incur a miniTTnTTn of tardiness records in arriving in 
those classes. Directions for administration and scoring are provided. 
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ATTITUDE TOWARD SCHOOL (continued) 



Grade Level Completion (Secondary, Grades 11 and 12) 

The percentage of students enrolling in school each semester who 
complete that semester may be computed from school records. In 
general, those students who leave school prior to graduation tend 
to be those with more negative attitudes toward high school. Direc- 
tions for administration and scoring are provided. 

School Conduct: Compliance With School Rules (Secondary) 

« 

Records are kept by school authorities to whom students have been 
referred. This observation indicator is based on the concept that 
students possessing favorable attitudes toward school will tend to 
accept the school rules and abide by them. Directions for adminis- 
tration of the School Conduct Record Form is provided. 

Unwillingness to Transfer (Secondary) 

This observation indicator involves presenting the student with an 
option to sign up for a possible transfer to a new class section. 
The holding power of a class (or school) has been found to be a 
correlate of positive attitude toward school. Directions for admin- 
istration and scoring are provided. 

The objectives and assessment measures included in this IOX collection were 
developed by the Instructional pbjectives Exchange with support from a con- 
sortium of Title III program representatives from 40 states. The present 
revised version was prepared following a field test with approximately 1,230 
students. Items underwent extensive subject matter reviews by subject area 
specialists, evaluators and teachers. Statistical analyses were conducted 
and test/retest and KR-20 reliability coefficients were reported. 

This 183-page IOX collection of measurable objectives is available from the 
Instructional Objectives Exchange, Box 24055, Los Angeles, California 90025. 
The IS 31-82 catalog price is $12.95. In addition, a set of the attitude 
toward school measures is available in spirit master form at the elementary 
level and at the secondary level. The listed price for each set is $29.95. 
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CALIFORNIA ACHIEVEMENT TESTS 
(CAT), 1977-78 



Horn- Referenced: Grades 7.-12 



CAT ia a nationally normed, standardized achievement cesc baccery. It was 
designed Co provide boch norm-referenced and criterion-referenced informa- 
tion for educational decision-making, leading to improved instruction in 
the basic skills areas. 

CAT includes 10 levels for the following instructional grade spans: 
10 (K.0-K.9); 11 (1.0-1.9); 12 (1.6-2.9); 13 (2.6-3.9); 14 (3.6-4.9); 
15 (4.6-5.9); 16 (5.6-6.9); 17 (6.6-7.9); 18 (7.6-9.9); and 19 (9.6-12..9). 
Two alternate forms (C and D) are available for Levels 13-19; Levels 10-12 
are available in Form C, only. Test include: Prereading (Listening for 
Information, Letter Sounds, Letter Forms, Letter Names, visual Discrimina- 
tion, and Sound Matching); Reading (Phonic Analysis, Structural Analysis, 
/ocabulary and Comprehension); Language (Expression and Mechanics); 
S pelling ; and Reference Skills . 

Cest administration time across levels ranges from 45-96 minutes for 
leading; 28-60 minutes for Mathematics; and 12-38 minutes for Language, 
lidpoint reference dates for test administration are November 3 and May 4. 
Information on test development, validity, and reliability, is reported in 
technical Bulletin I. 

lAT features include: Locator Tests ; Practice Tests ; expanded standard 
icore for use in functional out-of-level testing; interpolated percentile 
ank and NCE scores within the compliance period for Model Al ; standardized 
irections and timing for simultaneous administrations of different cest 
evels; Scoreze (self-scoring) Answer Sheets ; Student Diagnostic Profile 
heets for recordingindividual results, including Objectives Mastery 
cores; Class Record Sheets for recording group results; and the Class 
anagqment Guide , which provides follow-up instructional activieies. 

est Review Kics are available from the publisher: CTB/McGraw-Hil 1 , 
el Monte Research Park, Monterey, California 93940 (Telephone- 
08-649-8400 or 800-538-9547) or the publisher's regional offices. The 
rice listed in the publisher's 1982 catalog for a Test Review :<ic is 
10.35 for Primary (K-3), or Intermediate (4-6), or Advanced (7-12), and 
19.50 for all grades; packages of 35 reusable cest booklets for each level 
re $18.20 for Reading or Mathematics and 332.90 for the complete battery. 
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COMPREHENSIVE ASSESSMENT PROGRAM 
ACHIEVEMENT SERIES 
(CAP ACH), 1980 



ss ?rsK-22 



CAP ACH is a battery of nationally normed, standardized achievement tests 
based on a comprehensive set of objectives. CAP ACH was designed to 
provide an evaluation system to yield both norm-referenced and criterion- 
referenced information in the basic skills areas of reading, mathematics, 
and language. In addition, some levels measure student performance in 
other areas. 

The battery is comprised of 11 test levels. Suggested instructional grade 

rTo ; 0 ^ el «'\^ , AC ^ le rJ are: ? (pre k_k - 5); 5 <*.o-i.3>s 6 u.0-2.5); 

n'Vc 8 (3 ;°- 4 * 5) ; 9 C4.0-5.5); 10 (5.0-6.5); 11 (6.0-7.5); 
12 (7.0-9.5); 13 (9.0-11.5) and 14 (11 and 12). Two parallel forms (A & B) 
are available for Levels 7-12. Levels 4-6 are available in one form only. 
Tests include: Reading (Vocabulary and Comprehension); Word Attack - 
Mathematics (Concepts , Computation, and Problem Solving)7T anguage (Spelling 
Capitalization and Punctuation, and Grammar); Reference an d Studv Skills- 
Writing ; Science , and Social Studies . ' ' 

Test administration time across levels ranges from 20-85 minutes for 
Reading; 20-65 minutes for Mathematics; and 20-25 minutes for Language. 
Empirical midpoint reference dates for test administration are October 15 
and April 23. Information on test development, validity, and reliability 
is reported in che Technical Manual, Forms A and 3 

CAP ACH features include: Locator Tests ; Practice Tests ; expanded standard 
score for use in functional out-of-Level testing; and the Pupil Record and 
Class Analyzer forma for recording test administration results. 

Test Review kits are available from: Scott, Foresman Tesr. Division 1910 
Ease Lake Avenue, Glenview, Illinois 60025 (Phone: 312-7^.9-3000) '"The 
price listed in the publisher's 1982 catalog is $9.65 for combined levels 
4-8 or 9-12 or 13-14; and $24-31.20 for a package of 35 reusable test 
booklets for each of levels 9-14. 
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COMPREHENSIVE TESTS OF 3ASIC :ioz-rn-?.e?e?encsd< "-~-~es >'-' 

SKILLS (CT3S), 1981-82 



CT3S is a nationally normed, standardized achievement cest battery, developed 
to measure achievement in the basic skills areas included in state and 
district curricular. In addition, some CT3S levels measure student per- 
formance in other areas. The tests were designed to provide both norm- 
referenced and criterion-referenced interpretation. 

CT3S includes 10 levels for the following instructional grade spans: 
A (K.0-K.9); B (K.6-1.6); C (1.0-1.9); D (1.6-2.9); E (2.6-3.9); F (3.6-4.9); 
G (6.6-8.9); H (6.6-8.9); J (8.6-12.9) and K (11.0-12.9). Levels A-C are 
available in Form U only; Level D and above are available in alternate 
forms U & V. Tests include: Reading (Reading Vocabulary, Reading Compre- 
hension, and Oral Comprehension); Visual Recognition ; Sound Recognition ; 
Language (Mechanics and Expression); Spelling ; Reference Skills ; Mathemat- 
ics (Computation and Concepts & Applications); Science ; and Social Studies . 

Test administration time across levels ranges from 45-70 minutes for 
Reading; 15-64 minutes for Mathematics; and 15-59 minutes for Language. 
Midpoint reference dates for test administration are October 14 and April 
29. Information on test development, validity, and reliability, is reported 
in Technical Bulletin 1 . 

CTBS features include: process/content classification of items; an expanded 
standard score for use in functional level testing; Locator Tests ; Practice 
.Tests; interpolated national percentile and NCE scores within the compliance 
period for Model Al ; Scoreze (self-scoring) Answer Sheets ; Student Diagnostic 
Profile Sheets for recording individual test results, including Objectives 
Mastery Scores ; and the Class Management Guide , which presents information 
about interpretation and use of test results in instructional planning and 
provides suggestions for instructional activities. 

Test Review Kits are available from: CTB/McGraw-Hi 1 1 , Del Monte Research Park, 
Monterey, California (Phone: 408-649-8400 or 800-538-9547) or from the 
publisher's regional offices. The price listed in the publisher's catalog for 
a Test Review Kit is $10.35 for Primary (grades K-3) or Intermediate (grades 
4-6) or Advanced (grades 7-12), or $19.50 for all grades; and $15.40 for a 
package of 35 reusable test booklets for Reading or Mathematics (Levels F-H 
J and K, only). ' ' 
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DIAGNOSIS: MATHEMATIC LZVIL 3 Criterion- Referenced: Grades 3-5 

DIAGNOSIS is an ins crucc ional support system comprised of a series of 
objective-based diagnostic cescs in spirit master form. Objectives of the 
DIAGNOSIS system are to enable teachers to assess broad areas of achievement, 
pinpoint specific learner difficulties, and identify prescriptive materials 
and act ivi tie3 for remedial ins true t ion. 

Mathematics Level B, which spans the mathematics core curriculum for grades 
3 through 8, consists of two learning Labs. Lab Bl covers whole number 
topic* (concepts, addition and subtraction, multiplication and division, 
and word problems) and the easier topics in gecrtietry and measurement. Lab 
B2 includes fractions, decimals, and related topics, and the more difficult 
geometry and measurement topics (graphs, statistics, and probability). The 
Labs are designed for use with instructional programs, and may also be 
adapted to other types of existing programs. 

In implementing the Lab, teachers quickly assess students' skills and 
understanding of mathematics through the administration of the one-page 
Survey Tests. Students whose scores indicate learning difficulties on the 
Survey Tests are administered Probes , short one-page diagnostic tests that 
enable teachers to determine students' learning difficulties. Each Probe 
includes from one to four sections, each of which may be administered 
separately. 

The section "Error Sources and Activities" in the Teacher's Guide is used 
to determine reasons for errors made by students. In addition, the Teacher's 
Guide presents instructional activities designed to help students overcome* 
the various sources of their errors. The Prescription Guides list supplemen- 
tary remedial materials for each Diagnosis learning objective. The guides 
correlate the objectives to major textbooks, workbooks, duplicating masters, 
and the publisher'3 supplementary materials. 

Alternative Forms 1 and 2 are available for each test. For each form, 
there are eight Survey Tests and 10 diagnostic Probes in Lab 31 and seven 
Survey Tests and 10 Probes in Lab B2. Lab Bl also provides Tests of 3asic 
_Skin_s, in spirit master form, for use in determining whether students have 
adequately learned the basic facts of addition, subtraction, multiplication, 
and division. 

The Labs include various other components. Answer SCevs for the Probes and 
Survey Tests are printed on cards. To facilitate scoring, the answers 
printed on the cards are aligned with students 1 answers on the Probes and 
Survey Tests. The Student Record Sheet provides a profile of individual 
student progress and the Class Chart provides a profile of progress of the 
entire class with respect to objectives. The Class Chart may also be used 
for organizing small instructional groups and for managing individualized 
instruction. 
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DIAGNOSIS: M1THZHATICS LZVZL 3 (continued) 

A review set of Diagnosis: Mathematics 3 is available for $6.4Q from the 
publisher: Science Research Associacas, Inc., 155 Wacker Drive, Chicago, 
IL 60606 (Phone: 800-621-1664) . The price for Lao 31 or 32 is $95 and 
for both Labs is $160, as listed in the publisher's 19S2 catalog. 
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GATES-MacGINITIS U^f^i, C^ss Z-Z2 

The second edition of che Gaces-MacGinic ie Reading Tescs is comprised of a 
bactery of nationally noroed, standardized achievement cescs. The cescs 
were developed co measure achievement in reading and Co provide guidelines 
for organizing and evaluating boch individual and class instruction. 

The baccery includes seven levels (Basic R and A-?) and chree forms. Forms 
1 and 2 are available for all levels, and Form 3 is available for Levels D 
and E, only. Tesc levels are in-level for che following inscruccional 
grade ranges: Basic R (1.0-1.9); A (1.5-1.9); 3 (2); C (3); D (4-6); ' 
E (7-9); and F (10-12). Across levels A-F, cescs include Vocabu lary and 
Comprehension. Basic R includes four subcescs ( Leccer Sounds , Letter 
Recognition, Vocabulary, and Comprehension, and a cluster of icems cacegor- 
ized as Miscellaneous). Normacive informacion on Basic R subcezcs is 
given descripcively — high, medium, and low. 

Tesc adminiscracion cime for Levels A-F is 20 minuces for Vocabulary and 35 
minuces for Comprehension. There is no cime limic for Basic R, however, 
cocal cescing cime is approximacely 65 minuces. Eaoirical midpoinc reference 
daces for cesc adminiscracion are: Occcber 15 for all levels (except Level 
A for grade 1), February 15 for Level A for grade 1; and May 15 for all 
levels. Information on cesc developmenC, validicy, and reliabilicy, is 
included in che Teacher's Manual (published separately for each level) and 
the Technical Summary . 

Features of the Gates MacGinitie Reading Tests include: an exoanded 
standard score (except Basic R) for functional out-of-level testing; 
supplementary out-of-level norms (except Level F); Decoding Skills Analysis, 
included in the Vocabulary tests for Levels A and B, wich a form Co help 
organize informacion; uniform direccions and cime limics chac allow simul- 
caneous adminiscracion of more chan one level (A-F) wichin che same class- 
room: Teacher's Manual, which provides informacion on scoring, incerpreca- 
cion and use of scores, and also, for Level C, guidelines for error analyses 
of che Vocabulary cesc resulcs; Sel f-Scorab le Answer Sheec : and che CLass 
Summary Record Sheec form. ' " 

An Examinacion Kic is available from che publisher: The Riverside Publishing 
Company, Three O'Hare Towers, 8420 Bryn Mawr Avenue, Chicago, Illinois 60631 
(Phone:' 800-323-9540 or 312-693-0040 in AL, HI and ID. As indicaced in che 
publisher s cacalog, che price of an Examinacion Kic for each level is $2.28 
and $12.36 for a package of 35 hand-scorable cesc booklecs for Levels R-£* 
(Levels D, E, and F may be used wich separace answers sheets). 
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INSTRUCTIONAL OBJECTIVES EXCHANGE ( IOX) Criterion-Referenced: Grades Z- 



The IOX has chree types of criterion-referenced materials available: 
Che IOX Basic Skill System, Tes c/Praccice Exercise Sets, and Measurable 
Objectives Collections. 

IOX BASIC SKILL SYSTEM 

The IOX Basic Skill Tests, included in this instructional system, are 
criterion-referenced measures of minimal competency in reading, writing, 
and mathematics. At the elementary level, tests are designed for minimal 
competency assessment at the end of grades 5 or 6, and at the secondary 
level, as high school graduation minimal competency measures. Each test 
has two alternate forms (A and B) . Administration time for each test is 
approximately 45 minutes. 

At the elementary level, the Reading test includes seven basic skills: 
comprehending word meaning, comprehending syntax, identifying details," 
identifying sequences, determining main ideas, using a dictionary, and 
rising common reference sources. The Writing test includes seven baric 
skills: spelling correctly, punctuating correctly, capitalizing correctly, 
using pronouns correctly, selecting complete sentences, and expressing 
ideas in writing — optional assessment of an actual writing sample. The 
Mathematics test includes six basic skills: performing basic calculations 
with whole numbers, fractions, and decimals; solving word problems requir- 
ing a single arithmetic operation and measurement units; and interpreting 
tables and graphs. Elementary level tests include five items for each 
basic skill. 

At the secondary level, the Reading test has five basic skills: under- 
standing safety warnings, completing forma and applications, using common 
reference sources, determining main ideas, and using documents to rake 
action. The Writing test has four basic skills: using words correctly, 
checking mechanics, selecting correct sentences, and expressing ideas in 
writing — optional assessment of an actual writing sample. The Mathematics 
test has four basic skills: performing basic calculations; and solving 
everyday problems requiring single arithmetic operation, formulas, multiple 
arithmetic operations. Secondary level tests include four or five items 
for each basic skill. 

Other components of the IOX Basic Skills System include: test/practice 
exercises for secondary level only, in spirit master form; Teacher's Guides , 
which include detailed skill descriptions, instructional guidelines, and 
skill supplements; Test Manual , which provides directions for test 
administration and information on test content, technical test development 
and scoring procedures; Planning Aids , which include booklets on program 
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INSTRUCTIONAL OBJECTIVES EXCHANGE (continued) 



planning and scaff development , and audio-cassette training capes; and 
Basic Skill Answer Sheets, suitable for hand or machine scoring. 

TEST/PRACTICE EXERCISE SETS 

These are sees of criterion-referenced measures, in spiric mascer form, 
designed for use in various ways: as practice exercises in class and as 
homework, as diagnostic cescs, progress monitoring devices, or end of 
instruction measures* Most sets contain 35-50 separate test/exercises 
which can be matched to instructional goals. Each one- or two-pag« 
practice exercise consists of 5 or 10 items which measure a specific, 
we 1 1-de f ined ob j ec t ive . 

Each test/practice exercise set includes: a boxed set of IPX te sts, 
available in two alternate foras; a Test Manual , which provides descriptive 
test information and directions for administration and scoring; a Scoring 
Guide; a leaflet, "Description and Use Statement , " which provides informa- 
tion on the development of the measures and describes possible uses of the 
tests for instruction and evaluation; and a packet of six Classroom Manage- 
ment and Program Evaluation Foras , with instructions for management or 
evaluation implementation. 

Test/Practice Exercise Sets in basic skills areas include: 
READING 

Word Attack Skills, Grades K-6 . There are 38 tests (44 spirit 
masters) in alternate foras A and B. The tests cover: vocabulary, 
recognition of sounds and letters, and letter and word 
pronunciat ion. 

Comprehension Skills, Grades K-6 . There are 40 tests (59 spirit 
masters) in alternate foras A and B. The tests cover: reading 
comprehension of main idea, conclusions, sequence, and context 
clues in text, as well as punctuation, syntactical structures, 
and affixes. 

LANGUAGE ARTS 

Mechanics and. Usage, Grades K-6 . There are 38 tests (41 spirit 
masters) in alternate forms A and B. The tests cover: capitali- 
zation; punctuation; abbreviation; hyphenation; bibliographic 
fora; envelope, letter, and invitation form; plurals; possessives; 
pronoun referents; degree foras; subject-verb agreement; irregular 
past participles; misplaced modifiers; and commonly confused words. 
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INSTRUCTIONAL OBJECTIVES EXCHANGE (continued) 



Wo rd Forms and Syntax, Grades K-6 . There are 42 cescs (43 spirit 
taascers) in aLcernace forms A and B. The cescs cover: pares of 
speech, verb cense and cime, cypes of verbs, comolece sencencas, 
functions of parts of speech in sencences, types of subjects and 
direct objeccs, sencence paccerns, sencence transformations, and 
types of clauses. 

Compositi on, Library, and Literacy Skills, Grades K-6 There are 
37 cescs (57 spirit masters) in alternate forms A and B. Tests 
cover: sentence precision, ouclining, paragraph development, 
paragraph transition, and types of paragraphs as well as alpha- 
betization, dictionary use, fiction and nonfiction, the Dewey 
Decimal System, and card catalog use. It also includes tests 'for 
imagery, figurative language, sound patterns, figures of speech, 
literary elements, and literary types. 

MATHEMATICS 

Sets and Numbers, Grades K-6. Form A contains 35 tests (38 spirit 
maseers) which cover sees, numbers, and racional numbers. 

Operations and Properties, Grades K-6 . There are 40 cescs 
(40 spirit masters in alternate forms A and B. The tests cover: 
addition, subtraction, multiplication, division, and combined 
operations . 

Numeratio ns and Relacions, Grades K-6 . There are 39 cescs 
(44 spiric maseers) in alcarnace forms A and B. The tests cover: 
numeracion, racios and proporcions, graphs, scaciscics and 
probabilicy, and logic. 

Measuremen t, Grades K-6 . There are 38 tests (47 spirit masters) 
which cover: monetary measurement; linear measurement; liquid 
weight, distance, time, race, area, and volume measurement; 
temperature measurement; pressure, density, and concentration 
measurement; and scale drawings. 

Geometry, Grades K-6 . There are 36 tests (45 spirit masters) in 
alternate forms A and B. The tests cover: points, Lines, pLanes, 
simple plane figures, curves, angLes, parallelism, perDendicularity , 
triangle similaricy and congruence, circles, segments ," polygons , 
solids, constructions, formula use, and geometric symbols. 

Elements , SvmboLism, and Measurement, Grades 7-9 . There are 43 
tests (53 spirit masters) in alternate forms A and B. Tests cover: 
sets, integers, rational numbers, real numbers, numeration, 
sentences and logic, and measurement. 
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INSTRUCTIONAL OBJECTIVES EXCHANGE (continued) 



Geometry, Operations, and Relations, Grades 7-9 . There are 43 
tesc'., (53 spirit masters) in alternate forms A and B* Tests 
cover: geometry (operations ?nd properties of planes, and solids), 
statistics, ratios and proportions, and graphs. 

Materials are available from the Instructional Objectives Exchange, 
Box 24055, Los Angeles, California 90025. Frices listed in the 1981-32 
catalog are: $2.50 for a sample test set of the I0X Basic Skill Tests- 
elementary or secondary level and $37,50 Co $42.50 for a package of 25 
reusable tesc booklets in one subject area; $29.95 for one form of the 
Test/Practice Exercise Sets or $50.00 for both forms; and $11.95 to 
$15.95 for an IOX Measurable Objective Collection. 
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IOWA TESTS OF BASIC SKILLS 
(IT3S), 1978 



Norm-Esferenced: Graces 'i-d 



The IT3S is comprised of a series of nationally normed, standardized 
achievement tests: the Primary Battery, for grades K-3, and the Multilevel 
Edition, for grades 3-9. The tests were designed to provide comprehensive 
assessment of student achievement in important areas of basic skills. 
Also, the Multilevel EdiCion includes tests in other areas.. The IT3S was 
developed to provide both norm-referenced and criteriou-referenced inter- 
pretation of test results. 

The IT3S Primary Battery has four levels (5-8) which are available" in one 
form (7) only. The Multilevel Edition has six levels (9-14) which are 
available in two parallel forms (7 & 8). Recommended instructional grade 
ranges for the ten IT3S levels are: 5 (K.l-1.5); 6 (K.8-1.9); 7 (1.7-2 6)- 
B (2.7-3.5); 9 (3); 10 (4); U (5); 12 (6); 13 (7); and 14 (8-9). Tests ' 
include: Listening ; Word Analysis ; Vocabulary ; and Reading or Reading 
Comprehension; Language or Language Skills (Spelling" Capitalization, 
Punctuation, and Usage); Work Study Skills (Visual Materials, and Reference 
Materials); and Mathematics (Concepts. Problem-Solving, and Computation). 

Test administration time for Levels 9-14 is: 15 minutes for Vocabularv; 42 
minutes for Reading Comprehension; 52 minutes for Total Language Skills; 65 
minutes for Total Work Study Skills; and 70 minutes for Total Mathematics. 
Test administration time for Levels 5-8 ranges from 16-25 minutes for 
Listening; 14-20 minutes for Vocabulary; 20-24 minutes for Word Analysis; 
34-45 minutes for Reading or Reading Comprehension; 20-47 minutes for 
Language/ Language Skills; 49 minutes for Work Study Skills; and 25-55 
minutes for Mathematics or Mathematics Skills. Empirical midpoint reference 
dates for test administration are: October 28 and May 2 for the Primary 
Battery (Levels 5-8) and October 30 and April 28 for the Multilevel Edition 
Uevels 9-14). Information on test development, validity, and reliability, 
is reported in the Preliminary Technical Summary . Teacher's G uide (one for 
each of Levels 5 & 6, 7 & 8, and 9-14), and the Manua l for School 
Administrators . ~ " — ■ 

Features of the ITSS include: Practice tests; uniform directions and time 
limits that allow simultaneous administration of more chan one level of the 
Multilevel Edition within the same classroom; NCE norms booklet; expanded 
standard score for use in functional out-of-level test administration; 
Teacher's Guide, which provides suggestions fdr improvement of students' 
skills in areas covered by the tests; and separate IT3S percentile norms 
for large cities, high and socioeconomic schools, and Catholic schools; 
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IOWA TESTS OF BASIC SKILLS (continued) 



publisher forms including: Student Resort Folders — How Are Your Basic 
S * lUs ' |g£i^ *J°h±* Chares, and Profile Chares for Averages: and pubiica- 
C "" : H ° v Co Uss c he IT3 S Co Imp rove Instruction , and Research that Built 
the Iowa Tests of 3asic Skills " ~~ 

An Examination Kit is available from the publisher: The Riverside Publishing 

2S?? n fcu ThrM 2 Ha " T ° WerS ' 8420 Bryn ^ Avenue, Chicago, Illinois 
60631 (Phone: 800-323-9540 or 312-693-0040 in AL, HI, AND IL) . As indicated 
in the_ publisher's 1982 catalog, the price of an Examination Kit is $6 e or 
the Primary Battery or the Multilevel Edition, and $39 for a package of 
separate test booklets for each of Levels 9-14. 
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MATHEMATICS IN OUR WORLD , 
SECOND EDITION, 1981 



Cz^itevion-Refarenced: K-3 



These object ive-referenced tests, all on duplicator masters, are part of an 
instructional system for mathematics. The tests, packaged in individual 
sets by grade level are available separately from the publisher. Each set 
includes a series of module tests; an end-of-year test; Teachers' Gui de and 
Answer Key ; and multiple-choice answer sheets. • 

A detailed scope and sequence chart categorizes the content areas covered 
in the textbooks and tests. Content areas include: Counting and Place 
Value; Addition/Subtraction/Multiplication/Division of Whole Numbers; 
Fractional Numbers; Decimals; Estimation; Measurement; Geometry & Graphing- 
Percent; Problem Solving & Applications; and Special Topics. 

For each grad« level, there is a series of instructional modules, each 
with specifically defined objectives. There is a test designed for each 
module to determine students' mastery of the facts, skills, and basic 
concepts of that module. - Many of the tests cover more than one objective, 
and the number of items that relate to a specific objective vary. Also, 
the timber of items on each test may vary— any where from about 6 to 26 or 
more, depending on the skill area. However, most tests are printed on one 
page. The item format is varied—some items are direct response items but 
most are multiple-choice, with three or four ODtions. On some tests, 
optional items are included. Some tests at the early grade levels include 
simple line drawings depicting children's activities. 

Testa for the Mathematics in Our World series are not norm-referenced, 
standardized tests. There are no strict time limits or formal test adminis- 
tration procedures required. The publisher advises that the module tests 
should be used pr'jarily as a diagnostic tool for determining which concepts 
or skills need further development. 

Components of the instructional system include: student text & teachers' 
r e , d \ C u 0r V DuplicaCor Mas^r Tests: workbook & teachers' edition; Enrichment 

Workbook; and the Teachers' Resource Book , which provides reproducible 

masters for reteaching and enrichment, in a loose-leaf format. Across 
levels, the sets of duplicator masters range in price from $24.15-$25 .86 
as indicated in the publisher's 1982 catalog. Additional information is' 
available from the publisher: Addison-Wesley Publishing Company, South 
Street, Reading, Massachusetts 01869 (Phone: 617-944-3700) . 
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MEASURES OF SELF CONCEPT, REV. ED., 1972 



Affective: Z-12 



This is an IOX measurable objectives collection of affective objectives and 
related assessment measures available in paperback book format. The book 
contains complete tests, along with description and rationale', directions 
for administration, and scoring guides. The self concept measures are 
intended for use in pretest/posttest evaluations of programs designed to 
improve or impede increasing negativism of students 1 self concepts. Users 
may select from among the 42 clearly defined objectives and related measures 
those whichthey consider to be appropriate for their instructional settings 
Local modifications to the measures may be made if certain items are 
considered inappropriate for a given educational setting. Items may be 
deleted, modified, or added. However', care must be taken chat changes are 
consistent with the objective to be assessed. The measures are designed to 
be used, for assessment of group attitude only and not for individual 
assessment . 



The objectives and related measures are arranged into three grade levels: 
Primary (K-3), Intermediate (4-6), and Secondary (7-12). The measures 
focus on four dimensions of self esteem: scholastic (derived from success 
or failure in scholastic endeavors), peer (associated with peer relation- 
ships), family (yielded from family interactions), and general (a comprehen- 
sive estimate). Also, measures are categorized by type; direct self 
report measures solicit student opinion in, a straightforward question-answer 
format; inferential self report measures permit inferences based on student 
response to indirect stimuli questions; and observational indicators permit 
inferences on the basis, of direct observation of student behavior. An 
overview of the measures included in the book follows. 

Direct Self Report Measures 



Self-Appraisal Inventory (Primary/Intermediate/Secondarv) 

This omnibus inventory which addresses all four dimensions of self 
concept: scholastic, peer, family, and general, is available in a 
separate version for each test level. Students demonstrate positive 
self concepts by indicating agreement with questions that reflect 
positive perceptions of the self in relation to school achievement, 
family, peers, and self in general; and by indicating disagreement 
with questions that reflect negative perceptions of self in these 
areas • 



The inventory includes 36 items at the Primary level, 77 at the 
Intermediate level, and 62 at the Secondary level. Scoring is 
obtained by assigning weights to each response as indicated in the 
directions. Administration time is about 20 minutes for the Primary 
level, 20-30 minutes for the Intermediate level, and 15-20 minutes 
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J2ASURSS OF SELF CCHCEPT, REV. SO. , 1972 (continued) 



How About ?ou (Incaraadiata') 

This inventory consists of 10 items, each wieh chraa aleemaeives 
that describe a parson in relation co school and school vorfc. The 
t student is asked co imagine ha is vricing an assay and co selecc 
the descriptions which bast describe him/her. The item alternatives 
reflect a continuum of success/failure behavior or perception of 
self. The inventory can be a dainia tared in 5-10 minutes. Relia- 
bility coefficients ara .63 (tast/retest) and .57 (5R-20). 

Word Choice (Secondary) 

This 19-ieem inventory presents the student with vord pairs con- 
sisting of adjectives that describe a person's general, personal, 
and social attributes. Each vord pair contains a semantic scale 
i bImm t'S'**' J?? ^erument is based on the assumption that 
a person vith a positive salf concept would imagine chac his/her 
peers perceive and describe him in a favorable vay. Administration 
anT.34 Oa-2of m€M ' R-UabUiC7 cs ««^«acs are .36 (cesc/recesc) 

For All I Know (Secondary) 

This -inventory consists of 10 hypothetical situations which describe 
Backgrounds for: achiavemert in school, scholastic intag-'-y 
confidence in school vorfc, scholastic initiative, and oehers. ' The 
SS* 1 r*a?onses include two positive behaviors and wo negative 
benaviors in mspecc to these areas. The assumocion is made chac a 
student with a positive salf concept vill perceive him/herself as 
successful and confidant in scholastic endeavors. Administration 
time is approximately 10-15 minutes. Reliability coefficients ar- 
.31 (test/retestt) and .76 (KR-20). 

Observational Indicators 

Ward Pagr'no. ^ rlmarv/Inta r mediata) 

S^w*"^ 3 **" Ctchai *» iavolTO creating a classroom sicuatiou ia 

' ^t^fflf 16 ; "*! PT0 ^f? d v±ca ch « °P clon oi placing their vorfc on 
the bulletin board or giving it tc cheir eeacher. The aeasur- Is 

vS? JZ^l T^'^ ■ e « taB «« *** * poaicive aalf ccnceoc 

will want to display their worfc. Directions for administration and 
scoring are provided. 

Perceived Approval Sicuaeion ffriaarr/Ingaraadiata/Saeondarv^ 

This assessment technique involves creating a contrived classroom 
situation in vhica a student vith high aelf concept vill identic 
him/herself as a member of a group that has von che approval 0 f che 
teacher. Directions for administration and scoring are provided/ 
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2SASCSES 0? SSL? CONCSPT, R£V. ED. , 1972 (concinued) 



Iafereacial Self Report Measures 
Televisioa Accors (Primary) 

This 18-ieea iavaaeory requeses scudeaes co ladicace chair 
williagness. co play a wlda variecy of rolaa la aa iaagiaary 
Calavisipa show. Tha assuapeioa made Is chac individuals who 
P °*^ aS * posieive sal£ concep'e will ba willing co pro j ace 
him/herself iaeo a widar variacy of rolas chaa oaa who has a 
lass scroag salf coacapc. SCadaacs rapl7 "yes" or "ao" co 
Cha quascioas. Adaiaiscracioa ciae is approximaeely 10 
aiaucas. Tha ease/reeese raliabiliey coefficieaC is .74 aad 
cha KR-2Q raliabiliey coafficiaac is .60. 

Tha Class Pi** fP-^- Hr/Iaceraediace) 

This insCruaeae asks cha scadaac co precead chac childrea are 
co ba salacead for a play and co salece chose roles for which 
his/her pears would choose him/her for. . The assuapeion made 
ia Chac an individual who has a posicive' self coacepc will 
percaive chac oehen would be likely co case hia/her in roles 
which carry a posiciva Image. For boch Primary and Iaeeraediaee 

1 f m i a \5 Ppr03daac ' 17 10 «• required for adjniaiseracioa 

of Che 20-ieaa iavaaeory. Tese/raeese reliabilicy coefficieacs 
ara .75 for cha Primary level aad .80 for cha Iaeeraediaee level. 
KR-20 reliabilicy coefficieacs are . 60 for che Primary level aad 
•78 for cha Iaeeraediaee level. 

Pareaeal Approval (Priaarv/IaceraediaCe) 

This iavaaeory attempts eo assess che exeeae eo which a child 
values him/herself as uacoadicioaally accepeed by his/her moeher 
daspica erivial or major misbahavior. The iaveacory iacludes 20 
ieeas oa che Primary level, which eakes abouc 10 aiauees co 
cospleee, aad 10 ieeas oa cha Iacarmadiaea level, which eakes 
5-10 aiaucas eo coaplace. Reliabilicy coefficieacs for che Primary 
laval ara .77 (ease/reeese) aad .55 OS- 20) . For ehe Iaeeraediaee 
level, coefficieaes ara .91 (case/recese) aad .73 (KR-20). 

Whac gould Y ou Do? (Inceraediace/Secondarv) 

This iaveacory praseaes 18 ficcicious sieuaeions, each follovr ; by 
four alcaraacive accioas. The sicuaeioas focus on che following 
dimaasioas; accoaodaeiag co ochers, expeceaeioas of accepeaace, 
courage co axprass opiaioas, williagaess so pareicipace, aad 
axpeccacioa of success. Tha number of posicive alceraacives 
seleceed coasieieuee a seudeae's score. There are 18 ieeas on che 
Iaeeraediace level aad 19 ieeas oa che Secondary level. Boch levels 
caka abouc 15-20 miauees eo adaiaiseer. Reliabilicy coefficieacs fo 
Cha Iaeeraediaee level are .64 (eesc/reeesc) aad .58 (Ka-20) and 
.69 (eesc/recese) and .78 (KR-20) for che Secondary level. 
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MEASURES OF SELF CONCEPT, REV. ED., 1972 (concinued) 



The assessment measures included in che book were d«veloped by che Inscruc- 
Cionai Objectives Exchange with support from a consortium of Tide III 
program representatives from 40 states. The instruments wers field tested 
with approximately 1,230 students in California. The items underwent 
extensive subject matter reviews by subject area specialists, avaluators, 
and teachers. Statistical analyses were conducted (cest/retesc and KR-20) 
and reliabilities coefficients ware reported. 

This 132-page I OX collection of measurable objectives is available from: 
Instructional Objectives Exchange, Box 24055, Los Angeles, California 90025. 
The price listed in the publisher's 1981-82 catalog is $11.95. 
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METROPOLITAN ACHIEVEMENT TESTS, C&tsrion-refeTeraed: Gvcczs Y.-s 

INSTRUCTIONAL BATTERY 
(MAT/ I) , 1978-79 



MAT/I consists of a series of eeses corresponding co the major instruc- 
tional goals of basic skills curricula chac were surveyed nationwide. The 
eases were designed eo provide informacion on ehe educaeional performance 
of individual seudcnes ia eerms of specific inseruccional objeceives. Each 
Inseruccional tese assesses one basic skill area. Inseruccional Baceery 
eeses were coordinaeed in concenc wieh ehose of ehe Survey Baceery. The 
ewo eese baeteries were nacionally normed and seandardized cogeeher, and 
have cercain psychoneeric equivalenes. 

MAT/I includes six eese levels for ehe following inseruccional grade spans: 
Primer (K.5-1.4); Primary I (1.5-2.4); Primary 2 (2.4-3.4); Elemencary 
(3.5-4.9); Incetmediace (5.0-6.9); and Advanced I (7.0-9.9). Two aleernace 
£oms (Jl & Kl) arc available for each level. Reading Inseruccional eeses 
include: Visual Discriminacion s Leceer Recognieion, Audieory Discrimina- 
cion, Sighe Vocabulary, Phoneme/Grapheme-Consonanes, ?honeme/Grapheme 
Vowels, Vocabulary in Concexe, Word Pare Clues, Race of Comprehension, 
Skimming and Scanning, and Reading Comprehension. Language Inseruccional 
eeses include: Liscening Comprehension, Punccuaeion & Capicalizacion, 
Usage, Grammar & Syncax, Spelling, . and Seudy Skills. Machemaeica Inseruc- 
cional cases include: Numeracion, Ceomecry & Measuremene, Problem Solving, 
Operae ions-Whole Numbers, Operacions-L»iws & Propercies, Operacions-Prac- 
eions & Decimals, and Graphs h Seaeiscics. Toeal Langu ^e and Tocal 
Maehemaeics scores are available. 

Test adminiscracion cime across levels ranges from 4-45 minuces for ehe 
Reading eases; 10-35 minuces for ehe Language eeses; and 20-30 minuces for 
Maehemaeics. Empirical midpoine reference daces for eese adminiscracion 
are Oceober 15 and April 20, for all levels. Informacion on eese develop- 
mene, validiey, and reliabiliey, is reporeed in ehe Teacher's Manual for 
Adminiseering and Incerpreeing- Inscruceional Baceery, published separaeely 
for each level, and ehe Mecropolican Achievemene Teses Special Reports . 

Peacures of^MAT/I include: Praceice Tescs ; Inseruccional Reading Level 
(IRL), a crieerion-referenced score which enables eeachers eo seleee 
appropriaee levels for eheir scudenes from among ehe major basal readers; 
expanded seandard score for funceidnal oue-of-level eescing; Class Record 
form; Teacher's Manual for Adminiseering and Incerprecing , which provides 
deeailed informacion on ueilizaeion of eese scores for inseruccional 
purposes; and ehe Mecropolican Achievemene Teses Special Reports . 

A Sampler kic is available upon requese from: The Psychological Corpora- 
Cion, 757 Third Avenue, New York Cisy, New York (Phone: 212-888-3500) or 
ehe publisher's regional offices. As liseed in ehe publisher's 1982 
cacalog, a compleee Specimen See for each level is 55, and a package of 35 
eese booklees for each level is $21.75 for Reading, $14.75-517.50 for 
Language Arcs, and $l8.75-$20.25 for Maehemaeics. 



Region V TAC 
ETS-MRC 



METROPOLITAN ACHIEVEMENT TESTS, 
SURVEY BATTERY 
(MAT/S), 1978-79 



Horn-Referenced: Grades Z-22 



MAT/S is * battery of nationally normed, standardized achievement eases 
designed eo measure performance in eha basic skills areas and, for some 
grades, in other areas. Survey Battery eases ware coordinaeed in content 
with ehosa of ehe Inseruceional Baeeery. The ewo eese baeeeries ware 
nationally normed and standardized eogeeher, and have certain psychomeerie 
equivalanes . The Survey cases also yield criterion-referenced daca, 
including estimates of Instructional Reading -Level. 

MAT/S includes eight test levels for the following inseruceional grade 
spans: Preprimer (K.0-K.5); Primer (K.5-1.4); Primary i (1.5-2.4); 
Primary 2 (2.5-3.4); Elementary (3.4-4.9); Intermediate (5.0-6.9); 
Advanced 1 (7.0-9.9); and Advanced 2 (10.0-12.9). The Primer through 
Advanced 2 levels are available in two alternate forms, JS & KS; ehe 
Preprimer level is available in form JS only. Teses include: Reading 
(Preprimer only), Reading Comprehension, Language, Maehemaeics, Science, 
and Social Seudies. ' 

Test administration time across levels ranges from 30-45 minutes for 
Reading Comprehension; 50 minutes for Reading (Preprimer); and 25-40 
minutes for Mathematics and Language, Empirical midpoint reference dates 
for ease administration are October 15 and April 20, for all Levels. 
Information on test development, validity, and reliability, is reported in 
C J* Teacher s Manual for Administering and Interpret ing- Survev Baeeery, and 
the series of Metropolitan Achievement Tests Special Reports ." 

Features of MAT/S are: Practice Tests : Inseruceional Reading Level (IRL), 
a crieerion-referenced score which enables teachers eo selece appropriate' 
levels for eheir students from among ehe major basal series; expanded 
scandard score for funceional oue-of-Level eeseing; Class Record form; ehe 
Teacher s Manual for Administering and Interpreting , which provides de- 
tailed information on utilization of test scores for instructional purposes- 
and the Metropolitan Achievement Tests Special Reports . 

A Sampler kit is available upon request from: The Psychological Corpora- 
tion, 757 Third Avenue, New York City, New York 10017 (Phone: 212-888-3500) 
or ehe publisher's regional offices. As liseed in ehe Publisher's 1982 
caealog, a eompleee Specimen See for. each level is $5, and a package of 35 
S12 C 75°9l4 tt 75 & * *** UV " 1 ' aeadintI or M «hemaeics, ranges from 
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NORTHWEST EVALUATION ASSOCIATION 
(NWEA) ITEM BANKS 



Itm Sank: Grades Z-12 



The purpose of ehe NWEA Itea BanJca is tJ fecilitaee che conacruceion 

!f J** rT - *! CMC " f0r aM in in «««ctioa*l plaaniug end program evalua- 
tion. Each of ehe ehree NWEA learn Banka: Raadiag, Uaguage Osege, aad 
Maehemeeice, includes: a coll.ctioa of ease item cards Agal r,L 
Cagalog, a Pacr'a Manual. «nd a ease cempiaca . The test ieem caris'ara 
indexed 10 ehe Coal Ic«m Ceealog, by goal aad Saseh laval. learns art ready 
for raprrjducei.cn following saloceiou and arraagemeoe. t 

The Reading Ieem Bank concaiaa aboue 1,050 items relaead eo over 150 goals, 
£J ch * «ac°«**cig» lean gaak ^^»« aboue 1,580 ieems relaead eo over 
220 goals. The uumber of teems and relaeed goals iacluded ia che mose 
raceaely developed Languag e Usage Item Bank are eo be made available. 

The User's Kaaual provides dec ailed iaformaeioa oa ehe use of che ieem bank 
aad Goal learn Caealog, including learn selcceioo and recrieval, aad ease 

Course Coals Colleceioas (execasivc eaxoaomins 'of course goals ia LangTIage" 
Arcs, Raadiag aad Language Osage, aad Meehcmaeics). Iaformaeioa oa 
arranging for eechaical coaeuleacioo oa ehe use of ehe NWEA learn Banks is 
iacluded ia ehe resource documeae, Cuide eo Coaauleanc Services . 

The ieem beaks were developed during aevea yeers of rasearch, ehrough ehe 
cooperaeive affores of educaeioaal ageacias ia Oregoo aad Washiageoo. ieems 
which were developed eo meee local diserice accds vera pooled, indexed eo 
curricular goals ideaeified ia ehe K-12 Course Coala Collections . . ii„ k ad 
end field ceseed, seaeiseically analyzed, aad Rasch calibraeed. The K-12 
gurse Coals CollecCions were developed by ehe Tri-Councy Coel Development 
Project, e consoreium of aboue 55 school diserices ia Oregoo, as a resource 
tor selecting seudeae learaiag ouecomes for use ia educaeioaal planning 
and evaluaeioo. r * 

A^^! r ,- af0 T e f 0a J" * v * il * bU fro ° **. Ray R. Millar, Horehwese Evaluaeion 

flUS ' W " hia * eon 94148 (phoa « : 2-6-242-9400). Each compleee ieem bank 
S500 (1982)* B8Ua86 *' ° r iUehw * cics) is «vailable for approximaealy 
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PRISM 



Item Bank: Grades 3-8 



PRISM is a s«ri«s of microcomputer software products designed for use in 
schools. The name PRISM is derived from its main functions: to print 
tests and dr il ls, to store records, and to manage Instruction and adminis- 
tration In the school setting. The software is designed for use with an 
APPLE 2, APPLE 2 Plus, or a Radio Shack Model 3 microcomputer. The soft- 
ware retjuires a microcomputer with 48k capacity, disk drive, and a 
compatible printer. 

PRISM Includes four criterion-referenced item banks from which users may 
generate customised "mastery, tests." Kach item bank Is organised, by broad 
skill areas, specific skills within these areas, relevant Instructional 
objectives, and related test items. To use the item bank, the user selects 
the area, determines the specific skills within the area, and identifies 
the objectives to be assessed. The user also determines the number of 
items to be Included In the test and the number of test copies to be 
printed. The first printed copy of the test contains the correct responses 
to all items. 

Each item bank Is available as a multi-disk program. Accompanying work- 
books include longer reading passages, diagrams, graphs, and other visual 
displays not reproduced by the printer. Also, the Classroom Management 
System (available on one diskette) provides teachers with a systematic 
procedure for monitoring student progress. A description of the four 
programs follows. 

PRISM MATH 1 is a criterion-referenced item bank tied to approximately 
200 instructional objectives common to mathematics curricular programs in 
grades 3 through 8. The item bank is divided into four major areas: 
Numeration, Operations, Applications, and Problem Solving. Each broad 
skill area Includes computer-generated items carefully selected for content 
by subject area specialists. The item bank was built around the 
Los Angeles Mathematics Program (LAMP), developed by the Los Angeles Unified 
School District and used as part of an Instruction program since 1975. 

The item pool consists of computer-generated items for all Operations 
problems and over 3,600 stored items. The computer program is stored in 
15 diskettes divided Into three levels: Level C-0 is for use in grades 3 
and 4; Level F for grades 5 and 6: and Level G-H for grades 7 and 8. At 
each level, the diskettes are: Program diskette, Numeration, Operations 
and Problem Solving, Applications, and Classroom Management Diskettes. In 
addition to constructing mastery tests, the item pool allows the teacher to 
select the particular Instructional objectives In which students need 
additional practice. A random problem generator can produce limitless 
drills for all operations areas: addition, subtraction, multiplication, 
and division. The package also provides teachers with a systematic 
procedure for -monitoring- student* progress. -PRISM MATH-1 is expected- to be 
available by late summer of 1982. 
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PRISM (continued) 



PRISM MATH 2 la currently in preparation. The criterion-refer encad item 
bank will focus on the basic and applied skills generally associated vita 
minimum competency tests. Designed for students in grades 7 and above, 
the program may be used to develop local miniawn competency tests and to 
provide remedial Instruction on an individualized basis. Items in MAIH 2 
will be ungraded. 

i 

PRISM READING 1 is a criterion-referenced, multi-disk program based on the 
.diagnostic-prescriptive approach to the teaching of reading. READING 1 is 
Intended for use In reading canters, resource rooms, and classrooms with 
any type of reading program. READING 1 is also intended for use in con- 
structing and monitoring Individualized Educational Programs. 

The Item bank, which includes over 2,000 items, is divided into three 
reading skills areas: Word Identification, Comprehension, and Study Skills 
that are typically taught in reading programs at grades 3-5. Each skill is 
measured by 8-16 items chat can be used for identifying instructional needs 
of students, practice and drill, and for assessing mastery. Items for the 
item bank were drawn from the Psychological Corporation's Skills Monitoring 
System for Word Identification and Comprehension. To these were added a 
bank of items for Study Skills. The three levels C, D, and E roughly 
correspond to grades 3, 4, and 5, respectively. The program is expected to 
be available by late summer of 1982. 

PRISM READING 2 is a criterion-referenced item benk focusing on the basic 
reading skills associated with typical nilnlTmm competency tests. It provides 
locally generated wfn-hnum competency tests as well as remedial drill and 
Instruction in the skill areas where competency does not meet the minimum 
requirements. The 1.900 items in the READING 2 item bank are ungraded. 
Important: comprehension skills such as reading for facts, using context 
clues, identifying cause and effect, and using reference materials form a 
substantial portion of the item bank. READING 2 is intended for students 
in grades 7 and above. The program is currently in preparation. 

For further information about PRISM, contact: The Psychological Corporation, 
737 Third Avenue, New York, New York 10017 (Phone: 212-888-3500) or the 
publisher's regional offices. The listed price in the publisher's 1982 
catalog is: PRISM MATE 1— $250 for Levels C-D, E-F, or G-H or $675 for all 
three levels. PRISM MATH 2 or READING 2 prices are not yet available. 
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Criterion-Referenced: Grades K-8 



READING YARDSTICKS (RY), 1982 (Norm-Refereneed) 



BY is a battery of criterion-referenced tests developed to measure student 
performance in the skill areas needed for reading mastery at each grade 
level—from kindergarten through grade 8. The tests were designed to 
provide diagnostic information on student mastery of specific instructional 
objectives. Also, norm-referenced score estimates for comparable subtests 
on the Iowa Tests of Basic Skills (ITBS) and the Tests of Academic Profi- 
ciency (TAP) are available through the publisher's scoring service.' * 

Reading Yardsticks has nine test levels (6-14). One form of a single level 
is available for each of grades K-8. Test levels are numbered to correspond 
roughly to chronological age. For example: Level 6 corresponds to age 6, 
and Level 14 corresponds to age 14. BY test levels, available in one form 
only, are in-level for the following grades: 6 (K), 7 (1), 8 (2), 9 (3), 
10 (4), U (3), 12 (6), 13 (7), and 14 (8). Mastery scores, expressed in 
percent of items answered correctly, are available for RY Parts (Skill 
Areas), Subtests, and Objectives. BY Parts and Subtests include: Discrim- 
ination: Discrimination/Studv Skills (Auditory Discrimination, Visual 

Discrimination, and Study Skills); Matching Letters and Words : Phonic 
•Analysis (Consonant Identification, Consonant Substitution & Variants, and 
Vowels)} Vocabulary : Comprehension : (Literal, Interpretive, Evaluative, 
Language, and Life Skills); Structural Analysis (Word Parts, Consonants, 
and Vowels); Study Skills (Reference Material, Organizational Study Skills, 
and. Pictorial Study Skills); and Reading Rate . 

Total administration time for each level ranges from 110-227 minutes. 
Guidelines are provided for administering the tests in two or three testing 
sessions, as appropriate. The equating study to equate raw scores on RY to 
raw scores on comparable subtests of ITBS and TAP was conducted in March 
and April of 1981 with a sample of about 5,000 students in grades K-8 and 
10. Empirical midpoint dates for estimated ITBS norms are October 28 and 
May 2 for grades K-3 (Levels 6-8) and October 30 and April 28 for grades 
3-9 (Levels 9-14). The midpoint reference dates for TAP norms (grades 
9-12) are October- 29 and April 21. Information on test development, 
validity, and reliability, is included in the Technical Re sore . 

Features of Reading Yardsticks include: the Teacher's Guide , which provides 
information on the structure and content of the tests, planning for test 
administration, interpretation of scoring service reports, and use of test 
results; various publisher scoring services; and a student eyesight check 
on Levals 10-14. 

An Examination Kit is available from the publisher: The Riverside 
Publishing Company, Three O'Hare Towers, 8420 Bryn Mawr Avenue, Chicago, 
Illinois 60631 (Phone: 800-323-9540 or 312-693-0040 in AL, HI, AND IL) . 
As indicated in the publisher's 1982 catalog, the price of an Examination 
Kit is $2.28 for Levels 6-8 or 9-12 or 13-14; and $26.40 for a package of 
35 reusable test booklets for each of Levels 9-14. 



13b 

5-33 



Region V TAC 
ETS-MRO 



SEQUENTIAL TESTS OF EDUCATIONAL Storm-Se ^ivenced: Grades ?reZ*22 

PROGRESS, SERIES III 
(STEP III), 1979 



STEP III is a nationally normed, standardized achievement eese battery. Ic 
was designed to measure studen t ac hievement in language arts, mathematics, 
science, and social studies. STEP III was developed for use in program 
evaluation and for diagnosing instructional needs. Norm-referenced domain 
scores, based on subscales of items within total subject area tests, are 
available. 

• 

STEP III consists of 10 test levels that were targeted to the following 
instructional grade spans: A (Pre K-K.5); B (K.5-1.5); C (1.5-2.3); D 
(2.5-3.5); E (3.5-4.5); F (4.5-5.5); C (5.5-6.5); H(6.5-7.5); I (7.5-10.5); 
and J (10.5-12.9). Subject area tests and domains for Levels E through J 
include: Reading (Vocabulary in Context, Literal Comprehension, and 
Inferential Comprehension); Vocabulary ; Writing Skills (Spelling, Capital- 
ization & Punctuation, Word Structure and Usage, and Sentence & Paragraph 
Organization); Listening (Listening Comprehension, and Following Directions); 
Study Skills (Dictionary Usage, Library Skills, and Reference Skills); 
Mathematics Computation (Addition of Whole Numbers, Subtraction of Whole 
Numbers, Multiplication & Division with Whole Numbers, Computation with 
Measures, Operations with Whole Numbers; Fractions/Decimals/Pe'rcents) ; and 
Mathematics Basic Concents (Numbers & Operations, Measurement/Geometry /Graphs, 
and Problem Solving); Science ; and Social Stud ies, Parallel forms X and Y 
are available for Levels C through J. 

Test administration time for Levels E through J tests is 40 minutes except 
Vocabulary, which is 20 minutes, Midpoint reference dates for test adminis- 
tration of Levels E through J are October 5 and May 10. Information on 
test development, validity, and reliability, is reported in the STEP Ma nual 
and Technical Report . 

SttS features include: Locator Teats : Practice Tests ; expanded standard 
score for use in functional out-of-level testing; NCEs; Grade Level Indicators 
(GUs), a grade level score based on actual test administrations in grades 
at, below, and above, the targeted grade span; standardised directions and 
timing for simultaneous test administrations of different test levels; 
Self-Scoring Locator Tests Answer Sheets; Local Scoring Class Record 
Sheets; and the publication, "The Next STEP: A Guiie to Test Taking and 
Test Use," which includes suggestions for follow-up instructional activities. 

Specimen sets are available from: Addison-Wesiey Publishing Company, South 
Street, Reading, Massachusetts 01867 (Phone: 617-944-3700). As listed in 
the publisher's 1982 catalog, the price of a specimen set for Levels S-J is 
$4.50; and from $11.50 to.$l5-for a package of- 35- reusable, -individual 
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SEQUENTIAL TESTS OF EDUCATIONAL PROGRESS, 
SERIES III, (continued) 



subject test bookltts. A package of 35 reusable Basic Assessment Test 
Booklets (Reeding, Vocabulary, Writing Skills, Mathematics Computation, and 
Mathematics Basic Concepts) is 924.50 for each level. For information 
after June 1983, contact the Director of Cooperative Tests and Services, 
Educational Testing Service, Princeton, New Jersey 08541 
(Phone: 609-921-9000). 
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SRA ACHIEVEMENT SERIES 
(SRA ACE), 1978 



Nam-SefeTeneed: Grades K-12 



^^STT^ standardized . chi . vtMnC 

«•« or perfora^c^on « US. If V i" 1 ° f ««valop- 

«h.e i, caughf^cLHliv ! J °* d C °" ,ni: * r8M ™P*«««aciv. of 

of tawIliSf ■SSJrSil.* 1 " ,M » ,M d,v « la P«« i " ».«ur. broad «r.as 

intcr^Lw^ , ,ighC M " l€V,U (A - B) - "landed in-lavl- 

Tesc adaiaiscracion eime across levels ranffas ^,,f " - 

&*adin»* /.*-«n - . renge* rrott 443-115 ninuCss for 

aeaaing; 45-50 minuets for Lanzuase Arcs* *mi 23-an * " 7 

Co achievement ctinn. u provide an introduction 

.c«ff *2S35 o? XL^cio^" 0 "' r providt 

cioaal planning \L StSS * ' lacarpracacion of scores, instruc- 

twaoc in n. Ir l2n 2! *»«?ra»eion and assistance, 800-621-0664, 

ZT2 ' * *?° C * at pric * li » C,d ia ch « Publisher's 1982 caeala. 

iLH: f ! r / coa P l,c « S P«««™ Sac for Mc h level and «J!l0 for all * 

Levels E-H. heading, Language Art?, and Mathematics) for 
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Norm-Referenced: Grades 1-3 

STANFORD ACHIEVEMENT TEST (Criterion-Referenced) 
(STANFORD), 1982 

This new edition of eh« Stanford Achievement Test is scheduled to be 
aval labia for the 1982-83 school year. Sttanford-82 is a battery of nation- 
ally noised, standardized achievement tests designed for the comprehensive 
assessment of the achievement status of students in the major skills areas. 
The tests were designed to provide both norm-referenced and diagnostic/ 
prescriptive information for use in instructional planning and administra- 
tive decision-making. 

Sttanford-82 has six levels for Che following instructional grade spans: 
w!*r, l a < l '**-'>i ***»*y 2 (2.3-3.9); Primary 3 (3.3-4.9); Intermediate 
I (4.3-3.9); Intermediate 2 (3.3-7.9); end-Advanced (7.0-9.9). Two parallel 
forms (E & F) are available for each level. Tests include: Sounds and 
T L ?"' g ? ' Word Study Skills Word Reading. Reading ComprehonaioTT v^hni^. 
Listening co Words and Stori.s, Listening Comprehension . Spell ingT TanT^gT/ 
English, Concepts of Number. Mathematics . Mathematics Computation^ Mathema- 
tics Applications, Environment. Science , and Social Science . An optional; 
holistically-scored, Writing test is available at the Primary 3 through the 
Advanced levels. A separate Using Information score, in the domain of 
study and inquiry, is derived from items in several subtests. 

Test administration time across levels ranges from 30-70 minutes for 
Reading; 70-93 minutes for Mathematics; and 30 minutes for Language. Tho 
faU "^P 01 " r«*«r«nce date for test administration is October 
7. The spring midpoint reference date will be available at a later date. 
Information on the national item tryout program is reported in Stanford 
Special Report No.l. Additional information on test development, validity, 
and reliability, will be reported in the Technical Data Report . 

Features of Sttanford-82 include: an expanded standard score for use in 
functional outt-of-level eeseing; week-of-eeseing; inter polated norms 
within the compliance period for Model Al; and NCE scores. Publications in 

Planni ng, and the Guide for 
Organisational Planning . * 

A Sampler kit for the Stanford Series is available upon request from: The 
Psychological Corporation, 737 Third Avenue, New York City, New York 10017 
(Phone: 212-888-3300) or the publisher's regional offices. For each test 
level, a complete Specimen Set is S6 and a paekage of 35 test booklets for 
Reading or Mathematics is $16, as listed in the publisher's 1982 catalog. 
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STANFORD DIAGNOSTIC READING TEST 
(SDRT), 1973-78 



Norm-Referenced: Grades 1-13 
' ( Criterion-Referenced) 



The SDRT is a baeeery of naeionally norm«d, standardized eases designed co 
naasura performance of eha major eompoaanes of eha raading proeass. Tha 
eases ara iaeaadad for usa in diagnosing pupils' serengehs and waalenessas 
in raading, and for planning inserueeionai seraeagias. 

Tha SDRT concains four ease lavals for eha following inserueeionai grada 
spans: Rad (1.3-3.3); Graan (2.3-3.3); Brown (4.5-9.3); and Blua (9-13). 
Two parallal forms (A & B) ara available. Domains and Subeases includa: 
Comprehension (Word Reading, Reading Comprehension— Seneences & Paragraphs, 
Lieerai Comprehension, Infereneial Comprehension); Decoding (Audieory 
Discriminaeion, Phonaeie Analysis, and SerucCural Analysis); Vocabulary 
(Audieory Vocabulary, Word Meaning, Word Pares); and Race (Reading Race/ 
Fase Raading and Scanning & Skimming). 

Tese adminiseraeion eima across SDRT levels ranges from 35-50 minuees for 
Comprehension; 21-70 minuees for Decoding; 20-24 minuees for Vocabulary; 
and 2-14 minuees for Raee. Empirical midpoint reference daees for ease 
adminiseraeion ara Oceober 8 and April 28 for eha Rad, Green, and Brown 
lavals (grades 1-9) and November 8 for ehe Blue level (grades 9-12). 
tnformaeion on esse developmene, validiey, and reliabiliey, is reporeed in 
ch « Manual fo r Administering and Incerprocina . published separaeely for 
each level. . 

Feaeures of ehe SDRT include: Progress Indicaeors (crieerion-referenced 
cue-off scores, for each concepe/skill domain, which indicaee ehe need for 
remedial inseruceion; Practice T««f (R«d aad Green levels); and Inserueeionai 
Placamene Reoore (for hand scoring); and Handbook of Inserueeionai Techniques 

and Materials, published separaeely for each level. In-level eeseing is 

recommended by ehe publisher, aiehough ehe SDRT has an expanded seandard 



A Sampler kie is available upon request from: The Psychological Corporaeion, 
737 Third Avenue, New York Ciey, New York 10017 (Phone: 212-888-3500) or 
ehe publisher's regional offices. As liseed in ehe publisher's 1982 caealog, 
a complete Specimen See for each level is 34, and a package of 35 test 
booklees for each level is S19 (Red, Green, and Brown) and $21 (Blue). 



score. 
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STAMFORD DIAGNOSTIC MATHEMATICS Dicgrtostia/PresarisHve : Grades *-2J 
TEST (SDhI), 1976-78 



Th« SDMT is a baecery of naeionally aoratd, standardized eases, designed 
for diagnoseic/prescripeive asse&smene ae eha beginning of an inaeruceional 
sequence. Tese adminiseraeion resules are ineended eo be used eo ideneify 
seudenes' inaeruceional needs and eo plan inaeruceional seraeegies. 

The SDMT has four ease levels for ehe following inseruceional grade spans: 
Red (1.6-4.5); Green (3.6-6.5); Brown (5.6-8.5); and Blue (7.6-13). 
Two aleernaee SDMT forms (A & B) are available. Teses include: Number 
Syseem and Numeraeion,. Compueaeion, and Applicaeions. 

Taae adminiseraeion eime acroaa ehe SDMT levels ranges from 25-30 minuees 
for Number Syseem and Numeraeion; 35-40 minuees for Compueaeion; and 30 
minuees for Applicaeions. Empirical midpoine reference daees for ease 
adminiseraeion are Oceober 8 and April 28, for all levels. Informaeion on 
ease development, validiey, and reliabiliey, is reported in ehe Manual for 
Admimseering and Ineerpreeing, published separaeely for each level. 

Feaeures of ehe SDMT include: Progress Indicaeors (crieerion-referenced 
cueoff scores) for each concepe/skiil domain, which indicaee ehe need for 
remedial ineeruceion; a Praccice Tese for ehe Red level; Inseruceional 
Placemene Repore (for hand-scoring; ; and ehe Manual for Adminiseering and 
Ineerpreeing, published separaeely for each level, which provides prescrlp- 
eive eeaching seraeegies. In-level eeseing is recommended by ehe publish- 
er, although ehe SDMT haa an expanded seandard score. 

A Sampler Icie is available upon requese from: The Psychological Corpora- 
eion, 757 Third Avenue, New York Ciey, New York 10017 (Phone: 212-888-3500) 
or ehe publisher's regional offices. As liseed in ehe publisher's 1982 
caealog, a compleee Specimen See for each level is $4, and a package of 35 
ease booklees for each level is $19. 
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TESTS OF ADULT BASIC EDUCATION 
(TABS) i 1976 



Achievement : Adult 
(Norm-Referenced) 



EMC 



TABE 14 a battery of basic skills levels developed primarily for assessment 
of uodereducated adults but intended for use in a variety of educational 

S !-?t° gS ;,N tABE . WM dMi «° td C0! <l> provide information abou; entry level 
skills, (Z) assist in planaiag appropriate instructional activities, and 
(3) measure growth in achievement after instruction. 

TABS' is available ia three levels: E (Easy), M (Medium), and D (Difficult) 
and two alteraate forms (3 and 4). TABE was derived from Levels Z-, 3, and 
4, of the California Achievement Tests, 1970 edition. Level E corresponds 
Co Level Z (grades Z.O to 4.9), Level 51 to Level 3 (grades 4.5 to 6.9), and 
Level D to Level 4 (grades 6.5 to 8.9). The language and context of CAT 
items were modified to reflect adult language and experience. Also, items 
dealing with abs trace language concepts included in CAT were kept to a 
minimum and applications of language skills were emphasized in TABE. As 
part of the development of TABE, a bias review study was conducted. 

Tests for all levels include: Reading (Vocabulary and Comprehension) and 
Mathematics (Computation, and Concepts & Problems). .Language tests 
(Spelling, and Mechanics & Expression) are available only for Levels M and 
0. Subtests are further divU*d into sections to permit evaluation of 
student performance in specific skills areas. 

The complete battery of tests for each level and form is included in a 
separate, color-coded test booklet, 3-1/Z by 11 inches. The format of test 
items is varied, bur. most are multiple choice with four or five options. 
For Level E, some items are presented orally by the examiner. Across TABE 
levels, Total Reading includes 8Z-85 items; Total Mathematics, 98-117 
items; Mechanics & Expression, 100-109 items; and Spelling, 3Z items. 

TABE includes the specially developed Practice Exercise and Locator Test, 
printed in the same test booklet. The Practice Sxercise7"c"onsisting-"o7" 
U multiple-choice items, was developed to provide students wich experience 
in taking tests and using a separate answer sheet. The Locator Test, a 
short screening measure, consisting of Z0 Vocabulary items and 18 Computa- 
tion ieems, was designed as a quick screening device to determine the 
appropriate TABE level for testing the student. The Practice Exercise 
takes approximately Z0 minutes to administer and the Locator Test 15 
minutes. The Practice Exercise and Locator Test can be administered and 
scored in less than one hour, for a- class of 30. 

Administration eime across the ehree TABE levels ranges from 48-59 minutes 
for Reading, 59-79 minutes for Mathematics, and 51-57 minutes for Language 
One proctor for every 15 students is recommended. The Examiner's Manua l 

one for each TABE level, includes the norms tables and also, provides ' 

directions for test administration, scoring, reporting, and interpretation 
of east results. Answer sheets available for each level include a Comolete 

Battery Answer Sheet and a Scoreze (self-scoring) Answer Sheet for each 

subject (Reading, Mathematics, or Language). 
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TESTS OF ADULT BASIC EDUCATION (continued) 



Scores include number correct (raw scores), expanded standard (scale) 
scores, and grade equivalents. The norms tables provide conversions of raw 
scores to scale scores and to grade equivalents, and scale scores to grade 
™P nom * b ** ,d 00 «1««ting data collected in Spring of 
1975. The equating sample included 18,183 students, representing 25 
^! e ^ Ct * J i 2.!. atatM » *i«iai«tered both the shortened version of 

CAT-70 and TABS. Grade equivalent norms for TABE were based on the scale 
score. Reliability coefficients (KR-20) for Total Reading and Total 
Mathematics ranged from .92 to .98. Other TABE components include: . 
Directions for Adminis tering the Practice 'Exercise and Locator Tm st- 
Stutfiutt Profile Sheets/ Analvsis of Learning Difficulties forToWifving 
instructional needs; and the Orouo Record Sheet for hand recording group 

in Basic Skills (LSI), a multi-level instructional support package available 
from the publsher. 

A Multi-Level TABE Examination Kit is available from the publisher 

S2lSa 5r *^? iU, oJ , i.S n «.??" arch F " rlt ' «° ac «*«y. California 93940 (Phone: 
408-649-8400 or 800-538-9547) or from the publisher', regional offices. The 
price luted in the publisher's 1982 catalog is $7.25. Peckages of 25 test 
booklets are 319.50 for each level. 
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WIDE RANGE ACHIEVEMENT TSST 
(WRAT) i 1978 



Norm-Referenced: ??9K-12-h 



Basically a clinical-type instrument designed eo assess achievement in 
basic skills, WRAT is frequency used as a screening esse eo determine ehe 
approximate instructional level of students. WRAT is a wide-ranged test 
aorm.do7.ge. WRAT is published in two test levels: Level I i. intended 
for use with 5- to 11-year olds and Lev.l II i, intended for those 12 years 
and older. Both lev.l«, priated on the sow four-p«g. test form, include 
three subtests: Reading (recognizing and naming letters and pronouncing 
words out of context) ; Sgellinj (copying marks resembling letters, aamT 
writing, and writing single words to dictation); and Arithmetic (couacing, 
readiag number symbols, solving oral problems, aad performing written 
computations). For Reading, level I has 100 items and Level II has 89; for 

22 ftllflZ 1 * 1 hMM 6 l r Cm f "* L,V€l 11 ha » 51 5 f « Arithmetic 
Level I has 63 items and Level II has 57. 

» 

The three subtests may be given in any order. The Reading subtest and the 
oral section of- the Arithmetic subtest are administered individually: the 
other parts of WRAT may be administered in groups. Students respond 
directly to the test form. The total testing time is about 20-30 minutes. 
Directions for the Examiner are presented ia the WRAT Manual. The reading 
aad spelliag word lists are available on both plastic cards and oa tape 
cassette. The latter can be used to traia examiners or in actual test 
administrations. 

Three maia types of scores are used to report test admiaistration results: 
grade ratiags, perceatiles, aad staadard scores. For each subtest, the 
number of items- the studeat answered correctly is transformed into a grade 
rating. (For the 1978 edition, the grade ratings are priated oa aa insert 
lacluded ia the WRAT Manual.) The grade ratiags are used to enter the 
appropriate age level norms tables ia the WRAT Manual to obtain percentiles 
and standard scores. The standard score is comparable to an 1Q or devia- 
tion score and is used for classifying students iato ability categories. 
The publisher cautions that grade ratiags should not be used to make 
comparisons amonj individuals or groups inasmuch as WRAT is aormed on see 
aad aot oa grade level. 

The aoraiag study iacluded 27 age groups (from ege 5 to ages 53-64) ranging 
in size from 400-600. The 1978 WRAT norms were adjusted on the basis of 
latelligeace tests which were also admiais tared to those ia the norming 
* Cud7 ' Manual presents technical information including split-half 

reliability coefficients for the 1965 edition which ranged from .94 to .98 
for the three subtests. 



144 



5-42 



Region V TAC 
STS-MR0 



WISE RANGE ACHIEVEMENT TEST (coneinued) 



Informaeion on obtaining « specimen see is available from ehe publisher: 
Jascak Assaciaecs, Inc., v526 Gilpin Avenue, tfilmingeon, Delaware 19806 
(Phone: 302-632-4990). Tha priea of a (pacinian sac including manual and 
plaeeic card* is 519.30 *nd a package of 30 ease forms is $8.75, as listed 
in eha publisher's 1981-82 catalog. M lxscea 
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Additional Criterion" and Norm-Referenced Tests 



DIAGNOSTIC MATHEMATICS INVENTORY (DMI), 1975-80 
(Criterion- and Norm-Referenced) 

The DMI Is a series of criterion-referenced tests that cover 
325 objectives found 1n both traditional and contemporary 
mathematics curricula currently 1n use. The DMI . 1s avail- 
able In seven levels (A-6) for grades 1-7+. The tests were 
designed to provide diagnostic Information to mathematics 
teachers for prescribing Individual and group learning ac- 
tivities and to provide group mastery Information to admin- 
istrators for needs assessment, planning, and evaluation. 
The DMI 1s keyed to 11 text book series. CTB/McGraw-H1l 1 , 
Del Monte Research Park, Monterey, CA 93940. 



KEYMATH DIAGNOSTIC ARITHMETIC TEST, 1971-78 
(C1rter1on- and Norm-Referenced) 

This 1s an Individually administered arithmetic test for use 
with students 1n preklndergarten programs through grades 6, 
with no upper 11m1t for remedial use. KeyM&th 1s comprised 
of 14 subtests organized Into three major areas: Content, 
Operations, and Applications. The test has only one wide- 
range level with 209 test Hems arranged sequentially 1n 
order of difficulty. Procedures for establishing basal and 
celling levels to determine which specific Hems to adminis- 
ter to students 1s provided. American Guidance Service, 
Publishers' Building, Circle Pines, MN 55014. 



NELSON READING SKILLS TEST (NRST), 1977 
(Criterion- and Norm-Referenced) 

The NRST 1s a battery of nationally normed tests 1n three 
levels (A-C) for use 1n grades 3-9. The tests were designed 
to measure student achievement 1n basic reading skills. 
Tests Include Word Meaning and Reading Comprehension, which 
can both be administered 1n one class period. Optional 
tests Include Word Parts, which permits diagnosis of speci- 
fic problems, and Reading Rate. The three NRST levels are 
printed 1n a single booklet which permits group-administered 
Individualized testing. Self-Mark Answer Sheets allow both 
hand-scoring and Hem analysis. The Riverside Publishing 
Company, Three O'Hare Towers, 8420 Bryn Mawr Avenue, Chi- 
cago, IL 60631. . . . 
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PEABOOY INDIVIDUAL ACHIEVEMENT TEST (PIAT), 1970 
(Norm-Referenced) 

The PIAT Is a wide-range screening test of achievement 1n 
basic skills and knowledge, designed for use 1n kindergarten 
through grade 12+. There are five subtests: Mathematics, 
Reading Recognition, Reading Comprehension, Spelling, and 
General Information. Since the 84 Items are sequenced 1n 
order of difficulty, students are tested within an appro- 
priate range of difficulty, based on a basal and celling 
level. Scoring 1s completed during test administration. 
American Guidance Service, Publishers' Building, Circle 
Pines, MN 55014. 



PRI READING SYSTEMS (PRI/RS, 1972-77 
(Criterion- and Norm-Referenced) 

PRI/RS Is a criterion-referenced approach to assessment and 
Instruction Incorporated Into articulated Instructional 
systems 1n reading and related language arts. Skill areas 
Include: Oral Language, Word Attack and Usage, Comprehen- 
sion, and Applications. PRI/RS 1s available 1n five levels 
which span grades K-9+. Both graded and multi-graded sy- 
stems are available. Materials Include keyed references to 
widely used basal text series. CTB/McGraw-H1 1 1 , Del Monte 
Research Park, Monterey, CA 93940. 



STS EDUCATIONAL DEVELOPMENT SERIES (EDS), 1976 
(Norm-Referenced) 

The EDS Is a battery of nationally normed tests designed to 
measure achievement and ability 1n grades 2-12 and also, to 
survey and report Interests and plans 1n relation to test 
results for consellng purposes. The tests are available 1n 
five levels: Lower Primary, Uppery Primary, Elementary, 
Advanced, and Senior. Achievement tests Included 1n the 
battery are: Reading, Language Arts, and Mathematics. Scho- 
lastic Testing Service, Inc., 480 Peyer Road, Bensenvllle, 
IL 60106. 



THE 3-R'S TEST, 1982 
(Norm-Referenced) 

This Is a battery of nationally normed tests of basic skills 
In 11 levels (6-18) for kindergarten through grade 12. Tests 
Include: Reading (Vocabulary, Comprehension, and Study 
Skills); Language (Capital irrtl^n, ~Sprl 1 tirgv P-unctua t-1 on , — 
and Grammar); and Mathematics (Computation end Problem 
Solving). The 3-R's Is available 1n different editions. 
Administration time for the Achievement Edition 1s about 100 
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minutes; and for the Class-Period Edition, about 40 minutes. 
Easy-score answer sheets are available. The Riverside Pub- 
lishing Company, Three O'Hare Towers, 8420 Bryn Mawr Avenue, 
Chicago, IL 60631. 



W000C0CK-J0HNS0N PSYCHO- EDUCATIONAL BATTERY, 1977 
(Norm-Referenced) 

This 1s a wide-range battery of nationally normed tests 
designed to measure achievement, ability, and Int'erest 
level. There are 10 achievement tests 1n Reading (Letter- 
Word Identification, Word Attack, and Passage Comprehen- 
sion), Language Arts (Dictation, Proofing, Spelling, Usage, 
and Punctuation), and Mathematics (Calculation and Applied 
Problems). Basal and celling rules are used 1n most sub- 
tests to limit the range of Hems that must be administered. 
A shorter version of the test--Woodcock Language Proficiency 
8attery Is available. Teaching Resources Corporation, 50 
Pond Par.k Road, Hlngham MA 02043-4382, 



WOODCOCK' READING MASTERY TESTS (WRMT), 1973 

The WRMT 1s a 400-1tem, wide-range reading achievement test 
which 1s Individually administered across grades K-6+. 
Diagnostic and Instructional Implications may be derived by 
analysis of performance on each of the five subtests: Let- 
ter Identification, Word Identification, Word Attack, Word 
Comprehension, and Reading Comprehension. By establishing 
basal and celling criteria, the number of specific Items to 
administer to Individual students may be determined. Ameri- 
can Guidance Service, Publishers' Building, Circle Pines, MN 
55014. 



WRITING PROFICIENCY PROGRAM/INTERMEDIATE SYSTEM (WPP/IS), 
1981 

WPP/IS 1s a criterion-referenced assessment and Instruc- 
tional system for the management of expository writing of 
students 1h grades 6-9. Writing skills are tested with a 
criterion-referenced test with 70 multiple-choice Items 
measuring 14 objectives, and three field-tested exercises 
which may be evaluated by holistic or primary-trait scoring. 
Teachers may use the results from both types of tests to 
evaluate students' strengths, Identify areas of need, and 
plan Instruction. CTB/McGraw.-H.1 1.1 .,. Del Mpnte Research Park, 
Monterey, CA 93440. 
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Additional Inventories - Affective and Other 



Burke's Behavior Rating Scales 
Western Psychological Services 

Career Maturity Inventory 
CTB McGraw-Hill 

Coopersmlth Self-Esteem Inventory 
University of California at Davis 

Intellectual Achievement Responsibility Inventory 
Fels Research Institute 

Kuder Occupational Interest Survey 
Science Research Associates, Inc. 

Locus of Control Scale for Children 
Educational Testing Service 

Minnesota Vocational Interest Inventory 
Psychological Corporation 

Minnesota School Affect Assessment, The 
Center for Educational Development 

P1ers-Harr1s Self-Concept Scale 

Counselor Recordings and Tests (Vanderbllt University) 

Strong-Campbell Interest Inventory 
Stanford University Press 

Tennessee Self-Concept Scale 

Counselor Recordings and Tests (Vanderbllt University) 

Vlneland Social Maturity Scale 
American Guidance Service 

Wide Range Interest and Opinion Test 
JASTAK Associates 

Work Values Inventory 
Houghton-M1f f 1 1 n Company 



Additional Item Banks 

California Department of Education 
c/o Dr. William Padia 
721 Capitol Mall 
Sacramento, CA 95814 
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Fountain Valley Teacher Support System 
c/o Richard ZweTg Associates 
20800 Beach Blvd. 
Huntington Beach, CA 92648 

Los Angeles County Test Development Center 

c/o John Martois 

9300 E. Imperial Highway 

Downey, CA 90242 

National Assessment of Education Progress 
c/o Jack Schmidt 
Suite 700 

1860 Lincoln Street 
Denver, CO 80295 



Sources for Further Information 

The mental measurement yearbook . Buros, O.K. (Ed.), 
Highland Park, NJ : The Graphon Press. 

Tests in print . Buros, O.K. (Ed.), Highland Park, 
nTi The Graphon Press. 

A sourcebook for mental health measures . Comrey, A. L . , 
Backer, t. e., & Giaser, E . n., Los Angeles: Human 
Resource Institute, 1973. 

Ths CSE test evaluation series . Hoepfner, R., et al, 
Los Angeles: Center for the Study of Evaluation, UCLA 
Graduate School of Education. 

An omnibus of measures related to school-ba sed attitudes. 
Princeton, NH: Educational Testing Service, Center for 
Statewide Educational Assessment, 1972. 

Measures of soda! psychological attitudes . Ann Arbor, 
ATI Survey Research Center, Institute for Social Research, 
University of Michigan, 1973. 
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Appendix 5-A 
A CR I TER I ON-REFERENCED TEST RATING SCALE 



JTricttons 

I. List In the space provided tht tit It, ltvtl, and form of 
each test btlng rated. 

!• Respond to tach question using tht following codt: 

Yts » ✓ No - - Unctrtaln • 0 

I. If tht question is not relevant, leave the space blank. 



I rest rttle. Level, andTor 



\. TEST CONTENT/VALIDITY 

1. Are the test 1 tarns clearly related to the specified test 
objectives? 



2. Dot* the set of objectives and test Items measured by the 
test natch the set of objectives taught by the program? 



3» If the test covers a variety of objectives, can a subset 
of selected objectives be attain 1 stared? 



RELIABILITY 

1* Is each objective measured by enough ttst Items to 
reasonably determine student mastery? 



2. Is the reliability for the subset of Items measuring each 
objective high enough? 



3. Is the reliability for the entire test high enough? 



4. For each objective. Is the criterion required for mastery 
(1«e«, tht number of Items answered correctly) set at a 
reasonable level? 



5. For the entire test. Is the criterion required for mastery 
set at a reasonable level? 



STUDENT APPROPRIATENESS 



1. Is the response form simple enough for the students to 
understand? 



2. Are separate answer sheets avoided for primary grades? 



3. Is administration tlmt of acceptable length for tht stu- 
dents? 



*• Kill students be able to understand the test Instruc- 
tions? 



Appendix 5-A (Cone.) 
A CRITERION-REFERENCED TEST RATING SCALE 



I. List in tht space provldtd the title, level, and form of 
each ttst being rated. 

I. Respond. to tach question using tht following code: 

Yts • 1 No • - Uncertain • 0 

3. If the question 1s not relevant, leave the spaee blank. 


I ifnwnLjmi'jjmi.r.ncKn; 








• 


mil sbuuvrib* aw coie cu ujioei vteno cne ronnat or tne 
test Items? 










*» •*»— rw«uin^j iev«i rwmnrwti 0/ cne test lvems appro* 
prlate for the students? 










7. Is the setting required by the test the type of setting 
1n which students In the program function best? 










6. Are the Hens of an appropriate interest level for the 
students? 










D. ADMINISTRATIVE CONSIDERATIONS 

1. Are teachers or staff who will be administering the 
test adequately trained and/or experienced 1n appro- 

pniM •uMini*vnwiun or die we»w' 










2. Is the cost per pupil acceptable within budget con- 

ft Mint*? 










3. Is the administration time required for the test of 
iccipMDii i*n^wn in relation to cne amount or time 
available for testing? 










»» possioie to use cms test ror other administra- 
tive testing purposes? 










5. Can several levels of the test be administered at the 
bine co a group or students? 










6. Can the same form and level of the test be used for 
pretest and posttest? 










i. SCORING CONSIDERATIONS 

1. Are the desired scoring options available? 










2. If machine scoring 1s desired, does the publisher offer 
scores and score conversions needed? 


1 









Appendix 5-A (Cont.) 
A CRITERION-REFERENCED TEST RATING SCALE 



J7T5cE7on^^^^^^^^™^^^^^^^^^^^^^^^^^™^^™ 

U List In tht spact provided the tit It, Itvtl, and form of 
each test being rated. 

2, Respond to each question using the following code: 

Yes • V No ■ - Uncertain ■ 0 

J, If the question Is not relevant, leave the space blank. 



ffs^ffTo^evo^TB^onn 



3. For machine scoring, 1s the publisher's "turn-around" 
tine acceptable for Instructional purposes or reports? 



4, If hand scoring Is desired,* 1s the scoring procedure 
clear tnough to avoid errors? 



5. For hand scoring, are the necessary score conversion 
process** clear enough to avoid errors? 



6. Art tht tables required for scoring routinely provided 
by the publisher? 



7. Art results summarized In terms of objectives? 



F. NORMS • NOTE: Tills section Is optional and applies only when the test also provides norms. 



1. Dots tht test havt tmplrlcal norms for great level(s) 
of studtnts In tht program? 










Z. Arc tht normlng groups rtprtstntatlvt of tht studtnts 
In tht program? 










3. Art tht norms fairly recent? 










4. Dots tht ttst havt tmplrlcal normlng dates within two 
weeks of the prettst datt? 










5. Dots tht ttst havt empirical normlng dates within two 
weeks of the postttst datt? 










6. Oots tht ttst havt an expanded scale score or out-of- 
level norms? 










7. If ttst norms art based on Individual administration, 
will that be ftaslblt given staff time available? 










8. Will the test be administered In the same type of set- 
ting upon which tht norms were based? 
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Appendix S-B (Contd.) 
A NORM-REFERENCED TEST RATING SCALE 



Directions 

I. List In tha space provided the title, level, ind form of 
each test being rated, 

?.. Rtspond to each qutstlon using tht following codt: 

Yes « J No - - Uncertain - C 

3. If tht qutstlon Is not relevant, leave tht space blank. 


Fast Title, Level, and Form 










km TEST CONTENT/VALIDITY 

lm Art most program objtctlvts measured by tht ttst Itm? 






m 




2. Art most ttst Items taught In tht program? 










3. Can a subtttt bt adilnlstarttf which specifically 
matches tht program? 










1. RELIABILITY 










1. Is ttst reliability high enough? 


$ 








2. Do tht subtests havt acctptablt rt 11 ability estimates? 










C. NORMS 

I. Dots tht ttst havt empirical norms for grade Itvtl (s) 
of studtnts 1n tht program? 










2. Art tht normlng groups rtprtstntatlvt of tha studtnts? 










3. Art tht norms fairly rtctnt? 










4, Oots tht ttst havt tmplrlcal normlng datts within two 
weeks of tht prtttst data? 










5. Dots tht ttst havt tmplrlcal normlng datts within two 
wttks of postttst date? 










6. Dots tht ttst havt an expanded sea It scort or out-of- 
Itvtl norms? 










)• STUOEIfT APPROPRIATENESS 

I, Is tht rtsponst form slmplt enough for the students 
I to understand? 











154 



Appendix 5-8 (Contd.) 
A NORM-REFERENCED TEST RATING SCALE 



H rations 

L. List 1n the space provided the title, level, 2nd form of 
each test being rated. 

5, Respond to each question using the following code: 

Yes » y/ No • • Uncertain ■ 0 

I. If the question 1s not relevant, leave the space blank. 



Fes^THe^eveT^n^or 



2. Are separate answer sheets avoided for primary grades? 



3. Is administration >Am of acceptable length for the 
students? 



4. U111 students be able to understand the test Instruc- 
tions 



5. Hill students be able to understand the format of the 
test Hems? 



€. Is the reading level required by test Items appro- 
priate for the students? 



7. Is the setting required by the test the type of set- 
ting in which students In the program function best? 



8. Are the Items of an appropriate Interest level for 
the students? 



ADMINISTRATIVE CONSIDERATIONS 

1. Are teachers or staff who will ba administering the 
test adequately trained and/or experienced in appro* 
prfate administration of the test? 



I. If test norms art based on Individual administration, 
will that be feasible given staff time available? 



3. W111 the test be administered In same t/pe of setting 
upon which the norms were based? 



4. Is the cost per pupil acceptable within budget con. 
stralnts? 



5. Is the administration time required for test of an 
acceptable length in relation to the amount of time 
aval labia for testing? 



15, 



5-53 



Appendix 5-8 (Contd.) 
A NORM-REFERENCED TEST RATING SCALE 



Jirtctions 

l« List In tht space provided tht title, level, and form of 
each test being rated. 

lm Respond to each question using the following code: 

Yes ■ V No ■ * Uncertain ■ 0 

I. If the question 1s net relevant, leave the space blank. 



6. Is 1t possible to use this test for other administrative 
testing purposes? 



7. Can several levels of the test be administered at the 
same time to a group of students? 



8. U1!l at least two-thirds of the program occur between 
pretesting and posttestlng? 



\ SCORING CONSIDERATIONS 

1. Are the desires scoring options available? 



2. If machine scoring 1s desired, does the publisher offer 
scores and score conversions needed? 



3. For machine scoring, 1s the publisher's "turn-around" 
time acceptable for instructional purposes or reports? 



4. If hand scoring Is desired, 1s the scoring procedure 
clear enough to avoid errors? 



5. For hand scoring, are the necessary scoring conversion 
processes c'oar enough to avoid errors? 



6. Are the tables required for scoring routinely provided 
by the publisher? . 
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Appendix 5-C 
A SUGGESTED METHOD FOR 
REVIEWING TESTS IN RELATION TO PROGRAM OBJECTIVES 



Following these d1 rectlons , 1s a two-part rating form which 
can be used to analyze test content 1n relation to program 
objectives for a particular grade span. The purpose of using 
this form 1s to help determine which of several tests best 
matches the program objectives 1n reading, mathemat1.cs, or 
language arts. Upon completion of this rating form, the user 
will be able to determine, for each test, the following: 



• how many test Items there are for each one of 
the program objectives; 

• the total number of program objectives measured 
by the test; 

• the total number of Items measuring the program 
objectives; and 

• the percentage of Items on the test which mea- 
sure the program objectives. 



Comparing this Information across tests will help to deter- 
mine which test best matches the program objectives. 



D1 rectlons 

These directions give a step-by-step procedure for using the 
Test Content Review Form. Clank forms, suitable for copying, 
are Included after the directions. An example, Including a 
completed review form, follows the blank forms. 

1. On the Test Content Review Forms (One and Two) 11st the 
Title, Level, and Form of each test which 1s being re- 
viewed. 

2. On Form Two, Column 1, enter the total number of test 
Items for each test being reviewed. 

3. Obtain a 11st of all program objectives, numbering each 
objective consecutively. Transfer the number and a brief 
one- to three-word description of each- objectl ve onto 
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Appendix 5-C (Continued) 



Form One. Use as many copies of the form as necessary 
1n order to 11st all relevant program objectives. THE 
FOLLOWING STEPS SHOULD BE DONE FOR EACH TEST BEING RE- 
VIEWED. 

.4. Review the test for Items corresponding to program ob- 
jectives. Place a tally mark 1n the box below an objec- 
tive each time a test Item 1s found which measures that 
objective. Do not Indicate that a test Item measures 
more than one object 1 ve • (That Is, 1f a test Item ap- 
pears to measure more than one objective, do not place a 
tally mark for each objective; rather, tally only the 
one objective whlcn 1t best measures.) 

. 

5. Once all of the test Items have been reviewed, count the 
number of program objectives measured by one or more 
test Items and enter this number on Form Two, Column 2. 

6. Total the number of Items matching the program objec- 
tives (the total number of tally marks) and enter this 
number on Form Two, Column 3. 

7. In Column 4, enter the ratio of Items measuring objec- 
tives (from Column 3) to the total number of test Items 
(from Column 1 ) • 

8. Calculate the associated percentage for this ratio (Col- 
umn 3 - Column 1) and enter this percentage 1n Column 5. 
COMPLETE STEPS 4-8 FOR EACH TEST BEING REVIEWED. 

Interpreting the Results 

The Test Content Review Form provides a process by which a 
test review team can look more closely at the Items 1n a 
test 1n order to determine how well the test measures the 
program's objectives. 

Most parts of this form are straightforward. After complet- 
ing Form One, the user will be able to determine how many 
test Items there are for each program objective. Clearly, a 
test should cover most, 1f not all, program objectives. The 
number of test Items per objective 1s also Important when 
comparing tests. A test with three Items per objective would 
probably be preferable to a test with only one Item per ob- 
jective. 

Form Two 1s basically a summary sheet. Column 2 Indicates 
the total number of objectives measured by the test. Again, 
the better test would be one that covers the most program 
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Appendix 5-C (Continued) 



objectives. Column 3 Indicates the total number of Items 
measuring objectives. It would be preferable to administer 
a test which does not Include many Items unrelated to the 
program objectives. TKe best Indication of this 1s 1n Column 
4 the ratio of Items measuring objectives to the total 
number of Items (also expressed 1n a percentage — a higher 
percentage Indicates that more Items are of relevance to the 
program objectives). 

In many cases there will not be a clear-cut answer as to 
which 1s the best test; 1t nay be a matter of trade-offs. 
For example, one test may assess all program objectives, but 
with only one Item per objective. .Another test may cover 
most of the Items and with more Items per objective. Keep 1n 
mind when using this rating form that the purpose 1s to look 
carefully at a test 1n order to determine how well the Items 
assess the program objectives. If one test 1s clearly not 
better, then a judgment will have to be made, based on the 
preferences of the review team. 
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TEST CONTENT REVIEW FORM ONE 



Program Objectives 



TALLY OF ITEMS TO OBJECTIVES 



TEST CONTENT REVIEW FORM TWO 



est 
Itle, 
evel, 
era 


1 

Total Nimber 
of Test 
Iteas 


1 — 

! 2 

Total NiAber 
of Objectives 
Measured 


3 

Total Nunber of 
Item Heasuring 
Obje lives 


.Ratio of: 4 
Items Measuring 
Objectives (31 
Total Nunber of 
Iteas (1) 


5 

Percent ■ 
Colunn 4 
tines 100 


i 














• 










































• 





EXAMPLE 



The County Institution was 1n the process of selecting a 
test of basic mathematics skills tn be used for Initial 
assessment purposes. The test would be administered after 
students were placed 1n the program 1n order to help 
determine which skill areas needed attention. Of major 
concern to the review team was the fact that the students, 
although typically 1n their teens, were functioning at a 
much lower Instructional level. Because the student's 
attitudes toward the test content could affect their score, 
the team wanted to select a test with content that woirld be 
of Interest to the older student, yet assess skills at the 
more basic level. 

After conducting a preliminary review of six tests, the 
selection team decided to eliminate three of the tests from 
consideration. One test covered content totally Inappro- 
priate to the program. The other two tested for basic 
skills, but the content was totally Inappropriate for older 
students. The team then decided to review three of the 
tests 1n a more extensive manner. The results of the review 
process are Indicated on the Test Content Review Form 
EXAMPLE. On the basis of Its review or the tests, the 
selection team came to the following conclusions about each 
test: 



• The CDA Test measured four of the five program 
objectives; each objective was measured by 
three or more test Items; and 84% of the items 
measured their program objectives. 

e The Denby Test measured all five of the program 
objectives: each objective was measured by 
three or more test Hems; and 72% of the Items 
measured their program objectives. 

e The Sequential Test measured all five of the 
program objectives; each objective was measured 
by three or more test Items; and 80% of the 
items measured their program objectives. 



Based on this Information, the selection team decided not to 
use the Denby Math Test the 72% figure Indicated that 
there were too many test Items that did not measure their 
program objectives. With the choice narrowed down to the 
CDA (84%) and the Sequential (80%), they decided to go with 
the latter. The Sequential, although 1t had a lower 
percentage, did measure all of the program objectives, while 
the CDA did not Include Items which would assess program 
objective 5. 
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EXAMPLE 

TEST CONTENT REVIEW FORM ONE 



tie. 
vel, 
m 


I, 
Add 
Two-Digit 


f 

2. 
Add 

With Carrying 


'rograi Objectives 
3. 

Subtract 
Two-Dlglt 


4. 

Subtract 
With Borrowing 


5. 
Basic 
Multiplication 
Facts 






TALLY OF ITEMS TO OBJECTIV 


ES 




A Test 
Basic 
th Skills 
vel 2, 
rm A 


/// 


m 


/// 


m 


! 


nby 

th Test 
rel 1, 

rm A 


/// 


/III 


/// 


/in 


//// 


juentlal 
th Skills 
rel 3, 
m B 


/// 


/II/ 


///• 


/0 


//// 
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EXAMPLE 

, TEST CONTENT REVIEW FORK TWO 



»t 

Itle, 
svel, 
»rm 


1 

Total Number 
of Test 
Item 


2 

Total Number 
of Objectives 
Measured 


3 

Total Number of 
Items Measuring 

Objectives 


A 

Ratio of: 
Items Measuring 
Objectives (3) 
Total Number of 
Items (1) 


5 

Percent • 
Column 4 
times 100 














rest 
isle 
Skills 
2. 

A 


19 


4 


16 


16 
20 


84 


f 

Test 
1. 

A 


25 


5 


18 


18 

25 


72 


ntlal 
Skills 
3. 

B 


* 

» 

25 


5 


19 


20 
25 

• 


80 


167 ; 
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Appendix 5-D 
DETERMINING WHEN TO TEST OUT-OF-LEVEL 



There are some occasions when the person planning the test 
administration feels that the publisher's recommended test 
level may not be appropriate for the student(s) taking the 
test. If there 1s some question as to whether the test 
level will be too easy or too difficult for the student(s), 
out-of-level testing should be considered. There are some 
guidelines which will help to determine ahead of time 
whether out-of-level testing should be used. 

■ 

First, there 1s a general RULE OF THUMB for existing test 
scores: 



• If a student gets less than 1/3 (one-third) of 
the test Items correct (floor effect), then 
(s)he should probably be tested at least one 
level lower on the same test series. 

• If a student gets more than 3/4 (three-fourths) 
of the test Items correct (celling effect), 
then (s)he should probably be tested at least 
one level higher on the same test series. 



There are six other steps, any of which might be taken to 
collect more Information on when to do out-of-level testing. 

1. Review existing standardized test data ob- 
tained 1n previous years from .other Chapter 1 
students 1n the same grade as those currently 
under consideration. Compare the average raw 
scores with the above rule of thumb. If the 
Chapter I students 1n the same grado demon- 
strated a floor or celling effect, this year's 
students may do so as well. 

2. Review last year's test scores for the stu- 
dent(s) under consideration. Again, compare 
the average raw score with the rule of thumb. 
If the students' last year's scores Indicated 
a floor or celling effect, they may have the 
same problem this year. 

3. Review the test results obtained from the test 
used for student selection. Applying the pre- 
viously stated rule of thumb should give an 
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Appendix 5-D (Continued) 



Indication as to whether the test was too dif- 
ficult or too easy. 

Use the test publisher's locator tests to de- 
termine the student's functional level. 

Review the grade level of the instructional 
materials selected for each student under con- 
sideration. The level of the Instructional 
materials selected for a particular student 
should give a good Indication of the student's 
functional level. Additionally, test manuals 
sometimes Include linkages between basal read- 
ing series* and test levels. 

Obtain teachers' judgments about the student's 
functional level. Include past and present 
teachers . 



170 



5-64 



Appendix 5-E 
TEST ADMINISTRATION CHECKLIST* 



D1 sections 

This checklist may be used to ensure that the test admlnls 
tratlon goes as smoothly as possible and 1s done correctly 
Place a check mark next to each step as 1t Is completed 
Following each of these steps will result 1n sound test ad 
ministration procedures. 



1. ORDER TEST MATERIALS: 

a) Relevant Information on number of booklets, levels, 

and forms collected. 
b) Delivery dates of test materials confirmed. 



2. SCHEDULE TEST PERIOD: 

a) Test schedule wltliln two weeks of normlng date. 

bj Adequate class test time scheduled. 

c) Test date not around holidays or half-days. 

d) Test date not during fire drills or other distrac- 
tions. 



3. PREPARE TEST ROOM: 

a) Test manual consulted to determine recommended stu- 
dent group size for test administration. 

b) Desks, rather than tables, 1n testing room. 

c) Adequate lighting and ventilation 1n room. 



4. PREPARE TEST ADMINISTRATORS: 

a) Administrator's manual studied to learn all test- 

spedflc directions. 

b) Need for keeping accurate time emphasized. 

_ e) Necessity of responding to students' questions 1n 

terms of test-format, not test-content, stressed. 

♦Adapted from '.he Rhode- Island slldetape, "A Systematic Pro- 
cess for Standardized Testing." 
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Appendix 5-E (Continued) 



DISTRIBUTE TEST MATERIALS: 

a) Materials counted when received: 

b) Transfer and storage of completed tests arranged. 



PREPARE STUDENTS: 

a) Students Informed regarding the following areas: 

t purpose of test; 

• areas to be tested; 

• duration of test; 

• date, time, and location of test; and 
t materials to bring to testing session. 

b) Practice tests, 1f available, administered to all 
students . 



ADMINISTER TEST: 

a) List of students to be tested made available to each 
test administrator. 

b) Necessary materials, Including extra pencils and 
scratch paper, available. 

c) "Do Not Disturb" sign placed on door. 

d) Students seated away from each other. 

e) Time allowed for student questions. 

f) Sample Items administered, 1f any. 

g) Student progress monitored to ensure that answer 
grids are being clearly marked, that students are on 
correct section of answer sheet or booklet, etc. 

h) Student questions responded to only in terms of test 
format* 

1) Each section of test, or entire test, administered 

1n time prescribed 1n manual, 
j) All test materials collected and counted at end of 

testing session. 

SCORE TEST: 

a) For hand scoring, test administrator's manual read 
for directions on how to score test and stencils 
used where appropriate. 

b) For machine scoring, all answer sheets carefully 
packaged and accurately marked with- all* necessary 
Information . 
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6 

RECORDKEEPING 
FOR PROGRAM EVALUATION 
AND MANAGEMENT 
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6. RECORDKEEPING FOR PROGRAM EVALUATION AND MANAGEMENT 
THE RELEVANCE OF COMPREHENSIVE RECORDS 



Recordkeeping 1s a critical aspect of the overall evaluation 
and management of a program, especially 1n situations In 
which students may enter or leave a program at variable 
times during the year and 1n which students may come Into 
the program with a wide range of backgrounds, capabilities 
and Interests. Comprehensive recordkeeping techniques can 
contribute significantly to a variety of areas, including: 



(1) Student Management -- The coordination of 
Individual and group progress through both 
the Instructional and non-1 nstruct 1 ona 1 
activities 1n the program 

(2) Short- and Long-Range Planning — The variety 
of planning and evaluation activities that 
are conducted 1n order to obtain Information 
which, when used to direct plans and changes, 
will result 1n the program becoming more 
responsive to student needs 

(3) Evaluation and Administrative Reporting 
Requirements All of the activities con- 
ducted 1n order to determine program effec- 
tiveness, as well as those necessary to meet 
She administrative reporting requirements 



TYP ES OF RECOI LS AND THEIR USES 



Clearly there are many types of Information which should be 
maintained for any program as well as a variety of methods 
for organizing the Information. This sec£lon will Identify 
some of the types of Information which should be maintained 
and describe some ways 1n which the Information can be used. 
The user will have t* select the most appropriate types of 
Information on whlen to keep records on the basis of his or 
her program needs. Comprehensive records are the key to 
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conducting a variety c* i*t1v1t1es, panging from providing 
Information for prograir tiement descriptions to providing 
data for program evaluation purposes. 

It should be kept 1n mind that sorce of the types of records 
discussed here may overlap with those required by the 
institution. Additionally, one should be aware of the legal 
requirements as to the length of time certain types of 
Information must be maintained and requirements regarding 
accessibility of files. 



Student Management 

There are a variety of types of Information which should be 
kept on each student, as well as on the entire group. In 
general, three categories of Information which should be 
maintained Include: (1) Individual student files; (2) indi- 
vidual student progress records; and (3) group progress 
records. Each of these types of records can contribute vital 
Information to the development and updating of Individual 
student .plans , the management of activities during the 
course of the day, and the evaluation of the program. 



Individual student files should primarily cc»U<n background 
information on the student, such as genera* V: >Mpt1ve In- 
formation, previous school experiences, p* . » id patl on in 
special programs and services, test results (achievement, 
Interest batteries, attltudlnal surveys, etc.), and other 
general Information which will contribute to the development 
of the most effective Instructional plan for that student. 
On the basis of the Information 1n the file, the staff 
should be able to address the following types of questions: 



• What non-instructional services, such as 
medical or counseling, are indicated? 

• On the basis of the student's strengths, in- 
terests, etc., what short- aid long-term goals 
might be appropriate? What specific objectives? 

• What further testing 1s indicated? 

• What social situations should be encouraged? 
Avoi ded? 



A form for collecting much of the .i nf ormat 1on which should 
be maintained within the individual student file is included 
1n Appendix ti-A. 
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Clearly the Individual student files will vary 1n the amount 
of Information actually available. While some student files 
may contain virtually no Information, files on students who 
have been 1n the system for a longer time period may Include 
a wealth of data. Most Important, however, 1s that a system 
for collecting and maintaining this Information be devel- 
oped. The files should be maintained 1n a central location 
and updated on a periodic basis. If, for example, achieve- 
ment tests have been administered recently but the staff 1s 
not aware of the existence of this Information, then 1t 1s 
of no use. Finally, because much of the Information Is of a 
confidential nature, thes* files should be stored 1n a loca- 
tion not openly accessible, 

Individual student progress records should include a 
description of the student's individual plan, Including 
goals, a 11st of the objectives necessary to reach these 
goals, Information on the student's learning style and 
preferences, the planned dally activities and schedule for 
the student, any behavioral systems established with the 
student, and a progress chart. These records should provide 
the Information necessary to monitor the student's activi- 
ties on a dally basis, to provide a periodic review of 
student progress, and to make revisions 1n the student's 
program as needed. 

Ideally, this Information would be kept 1n a location easily 
accessible to the staff Involved and to the student. Up- 
dating parts of these records, such as the progress chart, 
could be a rcsponslbl Hty taken on by the student and moni- 
tored by the Instructor or an aide. For the staff, Informa- 
tion In these records can answer the following types of 
questions . 



e Where should the student be at this point 1n 
time? 

t Is the student progressing at a satisfactory 
rate? 

• What specific goals and objectives havs been 
met by the student? 

• What areas are causing the student difficulty? 

• What types of -actl v1 ties seem to work best for 
this student? Are not most effective? 



For the student, Information 1n these records can answer a 
variety of questions, Including: 
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• What objective should I be working on? 

• Am I making any progress? 

• What learning activity should I choose? 

• What are my responsibilities? 



By being Involved 1n the maintenance of progress records, 
the student 1s encouraged to take more responsibility for 
his or her actions. Knowing what objectives and activities 
are expected and how these fit Into both short- and long- 
term goals helps the student see how learning something that 
ma/ initially seem unimportant can lead to the attainment of 
more relevant goals. Tracking one's own progress will, 1n 
many cases, be a motivational activity 1n and of Itself, 
particularly 1f the student 1s on some kind of a reward 
system for demonstrating mastery of certain cognitive 
behaviors or other skills. Furthermore, learning to take 
responsibility for one's actions 1s a skill which will 
benefit the student 1n later school, work and life situa- 
tions. Examples of forms for maintaining Individual student 
progress Information are Include** 1n Appendix 6-B. 



6 roup progress records should Include a comprehensive 11st 
of tne goais and objectives taught by an Instructor or, 1n a 
certain course, the activities available to teach each 
objective, and a master progress chart by objective. These 
records should be Maintained regardless of whether students 
are grouped Into classes or provided with Instruction on a 
pullout basis. These records allow the Instructor to make 
decisions regarding activities which may benefit more than 
one student, to pair students for tutoring, or to decide 
upon other scheduling alternatives. Specifically, group 
progress records may help answer the following types of 
questl ons : 



e Which students would benefit from being sched- 
uled to participate 1n a certain activity, such 
as a counseling session over a defined topic, 
the administration of a special test, a field 
trip, or special tutoring on objectives which 
are causing difficulty? 

e Which more advanced students could be used as 
peer tutors and which students might benefit 
from being tutored? 

e Are there students who might form a small group 
to participate 1n a certain activity? 

17/ 
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Examples of group progress recording forms can be found 1n 
Appendix 6-C* 



Short* and Long-Range Planning 

The systematic maintenance of records will facilitate the 
conduct of the short- and long-range planning activities 
that are necessary to any program. The effectiveness of a 
program depends upon Its responsiveness to the needs of the 
students 1 n the program* Long-range planning encompasses 
everything that 1s done 1n order to ensure that the program 
best meets the needs of the students who make up the target 
audience. This Includes specific student services In both 
education and service areas, staff development, curriculum 
development, budget allocations, building and equipment 
needs, and follow-up services* Short-range planning encom- 
passes the activities carried out on a more frequent a-nd 
short-term basis to meet specific needs, such as student 
assignments to educational and non-educational services, de- 
velopment or revision of specific activities, assignments, 
resource allocations, and all other activities that are re- 
quired on a dally basis* 



Many of the student management records already mentioned 
contribute to both l° n 9- *nd snort-range planning activi- 
ties * Obviously* the individual student files and individual 
student progress records can contribute the basic Informa- 
tion necessary to schedule students for Instructional and 
non-1nstruct1ona1 services* Changing student needs and capa- 
bilities will Indicate where new materials should be added 
to the curriculum or what specific new activities need to be 
developed* Student goals and objectives will Indicate what 
type of staff are needed, both on a long-term and short-term 
basis* Individual student plans will similarly Indicate the 
types of building and equipment resources necessary to best 
meet needs; for example, on a short-term basis there may be 
a need for more programmed textbooks that seem to be very 
successful with the students* In the long-range picture, 
changing student needs may Indicate that certain vocational 
skills would be beneficial; these new skills may require the 
purchase of additional equipment or a certain type of 
facility* Clearly, the student management records can 
contribute to more than just the coordination of student 
activities* 

There are, however, additional re^sras which should be 
maintained for planning purposes, Including: (1) a 11st of 
goals, objectives and activities, cross-referenced by target 
audience; (2) an Inventory of consumable and non-consumable 
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resources; (3) a list of outside resources and types of 
services offered; (4) staff files; and (5) follow-up aca- 
demic, vocational, and job placement availability. 

A 11st of goals, objectives and activities cross-referenced 
by t a r g e t audience wTTI raci 1 1 tate FFe assi gnment o7 
sped f 1 c activities to meev Individual and group needs and 
1s also critical for coordination among courses in the 
Instructional plan, as well as for coordination with 
follow-up education and work placement. Ideally, there 
should be a master 11st which Identifies all goals addressed 
within a program. Each goal should be further described by 
the Intended target audience, the specific objectives 
covered, and. the activities available to reach the goals and 
objectives. Depending upon the size of the program, the 
number of students and staff Involved, the types of instruc- 
tional and n on -Instructional services available, and so 
forth, this Information may need to be classified by topic: 
academic, vocational, real wo rid survival skills, staff 
development, etc. 

In the area of short-range planning, having this information 
available will facilitate finding answers for the following 
types of situations: 



• A certain student 1s ready to learn a specific 
mathematics objective. What activities are 
already available to teach that objective? 

• A small group of students needs to learn how 
to behave during a job interv- w. Are there 
any simulations available to tich this con- 
tent? 

• Some new staff do not understand how to nego- 
tiate behavioral contracts. Are there any 
Instructional materials on file which can help 
them? 

• A student wants to .learn how to operate a cer- 
tain type of office equipment. Which objec- 
tives need to be mastered? 



The availability of this type of Information Is possibly 
more Important for larger programs, where Informal communi- 
cation 1s less likely. For example, a vocational Instructor 
teaching basic mathematics required to learn a certain skill 
may need additional activities which cover the math skills. 
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While in a small program the vocational Instructor may 
1 nteract Informal 1y with the ir- th Instructor to obtain 
activities, this may not occur so easily 1n a larger 
program. The Information available on goals, objectives, 
activities, and target audience 1s even more valuable for 
new or Inexperienced staff. 

In the area of long-range planning, this Information will 
help to address the following types of questions: 



• What types of academic and vocational programs 
are the students capable of moving Into 
successfully? 

• If a new program emphasis 1s going to be of- 
fered, in what areas will new activities have 
to be developed? 

• On the basis of the most frequently used mode 
of Instruction, what types of new equ1 pment 
might be purchased in the next two years? 

• How much repetition 1s there across skills 
taught 1n the business course and t^e English 
courses? 



An Inventory of cons u mable and no^consumable resources may 
seem , aT First , FT be a trivial type <TF record to be 
concerned with* However, without access to the necessary 
equipment and supplies, student ard i>taff activities may be 
severely Impeded, ultimately resulting 1n a reduction 1n 
program ef f ect1 veness ♦ It 1s therefore necessary to have an 
updated and accurate list which Indicates the status of 
consumable and non-consumable resources available for the 
conduct of the program, 

For short-term planning purposes, records on resources help 
answer the following types of questions: 



• Are the necessary answer booklets available to 
administer a certain test? 

• Are the resources available to conduct a spe- 
cial class 1n small engine tuning? 

§ The equipment repair person 1s due tomorrow. Is 
there any other equipment 1n need of repair? 

• Some funds are still available for supplies. Is 
there a particular Item that 1s required right 
now? 
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For long-term planning purposes, records on resources help 
answer the following types of questions: 



• Are the re any major budget allocations that 
need to be made 1n order to upgrade or replace 
existing equipment? 

• If a certain vocational training area 1s phased 
out during the next year, what amount of 
consumable resources will remain unused? 

• If the program enrollment Increases by five' 
percent next year, how much in the way of 
additional! funds would be needed for con- 
sumables? 

• On the bas^s of available repair records, 
should the sjme brand cf equipment be purchased 
next year or should : new vendor be selected? 



A list of outsltf r r?<.Qn rces and types of services offered 
will fl c 1 1 1 1 a t e £ t Tn g i eeds which cannot be met through 
services within the program. Depending uron the program size 
and Internal resources, ther$ may be areas or types of 
problems which cannot be adequately handled through the 
program. The staff must be ab?e to Identify outside re* 
soufc*s for help In meeting thftse needs. 

For short-term planning purposes, a 11st of outside re- 
sources and services offered w t: help solve the following 
types of problems: 



• A ,'iew student 1s badly in need of dental work. 
What community resources may provide the needed 
work? 

• Several teachers have Indicated to the coun- 
selor that a certain student seems to be very 
disturbed. The counselor, who 1s new to the 
community, wants to have the student assessed 
by an expert. What are the options for getting 
the necessary testing conducted? 

• An Instructor wants to build a science-related 
field trip Into an upcoming unit for a sma 1 1 
group of students* What community resources are 
ava's&ble for this type of activity:? 
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For long-term planning purposes, this type of Information 
can help answer the following types of questions: 



e Does the program staff have up-to-date Infor- 
mation regarding community resour.es which will 
provide physical or psychological assessments 
for students 1n the program? 

e Are community resources being adequately used 
by the program staff or do they need to be made 
more aware of t^o?. 1 * resources? 

§ Are there new p* ^rams 1n the community which 
could provide the program with useful re- 
sources? 



Records of staff data also provide valuable Input for short- 
and long-term planning activities. Again, the larger the 
program and staff Involved, the more Important this task be- 
comes. In addition to the required administrative forms, 
records should be maintained which describe any special In- 
terests or skills relevant to areas outside of the present 
assignment, courses or training sessions attended, requests 
for additional training, and so forth. Maintaining these 
types of records helps to answar the following types f 
questions which relate to both short- and long-term planning 
activities: 



• If a basic computer program 1s begun next 
year; are there 1n-house staff capable of 
assuming responsibility for or participating 1n 
the program? 

e There 1s a short-term need for an aide 1n the 
library skills course. Are there any aides with 
relevant background who could be freed up to 
participate? 

e One of the students has a special Interest 1n 
writing science fiction. Is there anyone on 
staff who could work with this student on an 
Individualized basis? 

• A training session 1s going to be offered on 
Implementing behavior modification techniques 
for disturbed adolescents. Who from our staff 
would benefit most from attending? 
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Rec ords describing follow-up academic, vocational and job 
pTacfcasnt opportunities are critical to the overall planning 
of student programs, the goals and objectives Included In 
the program, the development of activities, the coordination 
among courses 1n the Instructional program, and the coordi- 
nation with outside services and programs. Staff must be 
aware of the follow-up programs available to students, the 
entry-level requirements of these programs, how these pro- 
grams may be of benefit to the student, and where additional 
Information can be located. 

For short-term planning purposes, maintaining these types of 
records will help answer the following types of questions: 



© A student has been temporarily placed 1n the 
program, but really needs some job training. 
What are some options? 

• A student who 1s about to exit the program 
would really benefit from further counseling 
sessions. Is this service available? 

• A counselor wants to provide a select group of 
students with a preview of academic follow-up 
opportunities. Who can provide this type of 
presentation to th» students? 



For long ^erm pl.riifig purposes, maintaining these tj : ,f 
records can help answer the following types o< cr'Stu s: 



• For the student who will be placed 1n the aca- 
demic program at the high school, what mathe- 
matics, English, and study skills would be most 
beneficial ? 

• For the student who hopes to get accepted 1n 
the electronics training program at the voca- 
tional school, what are the entry requirements 
and skills which should receive emphasis? 

• A new work study program 1s being developed at 
the high school. Does our program teach skills 
wh1r;h will facilitate entry Into this program? 
What activities should b* added? 

• The support programs for the community job 
placement program are being reduced. How can 
our program compensate? 
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Evaluation and Administrative Reporting Requirements 



Evaluation Includes all of the activities conducted In order 
to determine the effectiveness of a program, as well as the 
various elements that make up that program. This Includes 
areas such as determining the effectiveness of: the program 
1n meetJng Individual student needs 1n cognitive and other 
skill areas; purchased or developed Instructional materials 
or activities; non-1 nstruet.1 ona 1 services; staff perfor- 
mance; staff training; the program in helping students to 
succeed 1n follow-up placements; budget expenditures; 
facility utilization; and any other program elements which 
warrant the determination of effectiveness. The systematic 
maintenance of records can contribute much 1n the .way if 
Information f or "eva 1 uat 1 on pu rpos es , thus simplifying the 
process of conducting the evaluation. Furthermore, the 
records maintained will provide most of tne Information 
necessary to meet the variety of administrative reporting 
requl rements • 

Many of the records already descrUed will contribute to 
evaluation and reporting activities. The student management 
files, Including the Individual student files, ths Indi- 
vidual student progress records, and the group prog- ass 
records can contain valuable Information to help answer the 
following types of evaluation questions: 



§ Are students 1n the program demonstrating 
growth as Indicated by standardized achievement 
tests? By mastery of program objectives? By 
locally developed criterion-referenced tests? 

e Mhat, are the general characteristics of the 
audience which Ihe program 1s serving? 

§ Are the non-1 nstructl ona 1 services responsive 
to student needs? 

e What types -f Instructional approaches and 
materials ara most preferred by the students? 
Least preferred? 

e How effective are the behavioral management 
techniques Implemented by the staff? 

• Based on posttest results, which activities 
seem to be most effective 1n teaching the 
objectives? Least effective? 

e KMch s'sff members seem to wor!: best with 
which types of students? 



Much of the information maintained for short- and long-tor* 
planning activities can also contribute to evaluation and 
administrative re porting requirements. However , the 
Information maintained in those records will more likely 
form the basis for background Information, as much of 1t 1s 
descriptive 1n nature* For example, the goals, objectives 
and activities cross-referenced by target audience form the 
foundation for evaluating program effectiveness. The on- 
going 11st of consumable and non-consumable resources can 
help determine the effectiveness of budget allocations. The 
11st of outside resources 1s a starting place to determine 
how well the program makes use of community resources. The 
records maintained on staff help determine whether staff 
training has been effective or how staff effectiveness can 
be Improved* Information on follow-up academic, vocational, 
and job placement programs available in the community can 
be a starting place to determine whether the program 1s 
realistic in regard to students 1 follow-up placement alter- 
natives. 

Naturally there are additional records which should be main- 
tained to meet evaluation and administration reporting 
requirements. These records should Include Items uch as: 
student attendance records; student selection procedures and 
documentation; results of past surveys administered to staff 
or students; follow-up data collected on students in aca- 
demic, vocational, and job placements; Interview documenta- 
tion with potential employers; results of post-evaluation 
activities; state reports; and all other Information which 
may form a basis for future evaluation or reporting activi- 
ties. Maintaining records in these areas can help answer 
the following types of questions: 



• Is there an unusually high absentee rate for 
certain types of students? 

• What are the most topical characteristics of 
students selected for the program? 

• Are students succeeding in their follow-up 
placements? What are their weaknesses? What 
are their strengths? 

• What would future employees like to see in the 
way of vocational skills from graduates of th'< 
program? 

18 o 
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Clearly there are many types of records which can contribute 
significantly to: student management; short- and long-range 
planning; and evaluation and administrative reporting re- 
quirements. As with any system, the Implementation 1s 
time-consuming. However, once a system for maintaining the 
necessary records 1s begun, the Information gained will be 
well worth the effort. 
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Person Completing Form 



Date 

Appendix 6-A 
INDIVIDUAL STUDENT FILE RECORD 

Student: ___ j ____ - ___ ____________ _____ 

(last name) (first) (middle) 

Birth Date: Parent/Guardian: 

* 

Age: Address: 



Last Grade 

Completed: Telephone: , 

Does the student have any special health problems, medication, etc.? 

yes no 

If yes, 

specify: . 



Date of entry Into the Program: 

Estimated length of stay: 

Has the student been 1n the Program before? yes no 

If yes, dates: , 

Location of last placement: • 

Type of program (e.g., academic, vocational, residential treatment): 



Dates attended: 



Who shou'd be contacted for further Information regarding this last 
placement (e.g., counselors, probation officer, program administrator, 
teachers): 

Name Name 



Telephone Telephone 
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Has the student received special services 1n the past (e.g., health care, 
counseling)? 

yes no unsure 

If yes, specify date(s) and type(s) of program: 



Has the student been placed 1n special types of programs (e.g., learning 
disabled, work-study)? 

yes no unsure 

If yes, specify date(s) and type(s) of program: ' 



TEST/OTHER INFORMATION AVAILABLE (MOST RECENT) : 
Achievement Test(s) ; 



Test Title, Form and Level: „ Mm _ m __ m ___ 

Date of Administration: Administered by: 

Area ■ Raw .-J-e . arcentile Standard Score 

Area Raw Scans ^ Percentile Standard Score 

Area Raw Score Percentile Standard Score 

Area Raw Score Percentile Standard Score 

Test Title, Form and Level: 

Date of Administration: Administered by: 

Area Raw Score ____ *-ercent1 - e _____ Standard Score 

Area Raw Score Percentile Standard Score 

Area Raw Score Percentile _ Standard Score 

Area Raw Score Percentile Standard Score 
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Test Title, Form and Leve 1 , r 
Date of Administration: — 

Area Raw Score 

Area Raw Score 

Area Raw Score 

Area 



Raw Score 



Psychological Assessment ($) : 
Name of Instrument: 



Date of Administration: 
Scores/Conclusions: 



Administered by: 



Percentile 
Percentile 
Percentile 
Percentile 



Standard Score 
Standard Score 
Standard Score 
Standard Score 



Administered by: 



Name of Instrument: 

Date of Administration: _____ Administered by: 

Sco res /Cone 1 u s 1 on s : 



Interest/Attitude Survey (s) : 

Name of Instrument: 

Date of Administration: Administer by: 

Scores/Conclusions: __________ 



Name of Instrument: ____ , 

Date of Administration: Administered by: 

Scores/Conclusions: 
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Other Assessments (Vocational. Aptltud 

Name of Instrument: 

Date of Administration: Administered by: 

Scores/Conclusions: ____ 



Name of Instrument: 

Date of Administration: Administered by: 

Scores/Conclusions: . 



Student's special Interests/hobbles: 



Student's job 1nterests/goaU;: 



How well does the student function with peers? 

With those younger? 

With those older? 

With adu Its/authority ? 
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How well does the student function Independently? 



In one-to-one situations? 



In small groups? 



In large groups? 



In highly structured situations? 



In unstructured situations? 



What are some potently methods of positive reinforcement which might be 
used with this student? , 



Are there certain s1tuat1ons/sett1ngs/types of Interaction which should be 
avoided for this student? 



What specific academic/ vocational strengths does the student have? 



191 

6-18 



What academic areas need special attention? 



What non-academic areas (Interactive skills, grooming, physical coordina- 
tion, etc.). need special attention? 



Has an Individual Student Plan been developed? 

Yes (attach to file) 

a No (If not, when will one be developed?) 

Individual (s) responsible for plan: 



The following Items should be Included In this file: 

• The student's schedule, both dally and planned special events (e.g., 
weekly counseling sessions). 

e A 11st of goals/objectives. 

• The Individual Student Plan. 

t Student Management Plans, 1f developed. 

• An Identification of special curriculum materials/programs being used. 

e Any Interest surveys, student preference questionnaires, etc., admin- 
istered by the program personnel. 

t Examples of past work. 

e Other past records/grades, which will be of use 1n planning the 
student's program. 
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INDIVIDUAL STUDENT PROGRESS RECORDS 



As discussed 1n Section 6, ti:ere 1s a variety of Information 
which can be maintained on Individual student progress. 
Three forms which may be of use are Included here. 



Form One Includes 
column to briefly 
objective there 1s 



a place for the student's name and a 
describe each objective. For each 
space to record the pretest date and 
score, notes regarding areas of difficulty, and a* short 
description of the Instructional materials assigned. There 
l?.2] so , a column for recording posttest results and areas of 
difficulty. As an example, 1n the abbreviated form below 1t 
can be quickly determined that the student passed the 
pretest for the first objective and 1s now working on 
another objective; the assigned materials are described 
the designated column. No posttest has been attempted. 



1n 



<nm. 



ft"***'- 1 



Utlraaiml JUbrl.U 



feet- 



M'JUMT'IBII II 1 1 HUM 



Form Two provides a method for collecting Individual 
progress Information on unit and enabling objectives. For 
each unit objective, a student's progress can be tracked 1n 
regard to each enabling objective, including test attempts, 
mastery date, and the test score. For example, 1n the 
u/ orm that f oUo**s, Unit Objective 1 has five 
enabling objectives. The student easily mastered Enabling 
Objectlvesl and 2, required two attempts for Enabling 
Objective 3, and 1s having difficulty with Enabling Objec- 
tive 4 the test has be.en attempted -three times and still 
has not been passed. 



the 
abbreviated 
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Form Three provides a way to collect Individual progress 
Information on both unit objectives and enabling objectives. 
For each unit objective there 1s space to record Information 
on both pretest and posttest scores. For each enabling 
objective there 1s a column to record test attempts, the 
date of mastery, and the test score. In the abbreviated 
form below the student did not pass the pretest for the 
first objective, but did go on to master the enabling 
objectives and the posttest. The pretest for the second 
unit objective was passed on the first attempt. The student 
did not pass the pretest for the third unit objective and 1s 
now working on the first enabling objective. 
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. Appendix 6-8 

INDIVIDUAL PROGRESS INFORMATION - FORM ONE 



ENT: 






:tive (Grief 
script Ion) 


Date 


ItSl 

Score 


of Difficulty 


instructional Materials 
Assigned (Brief Descrlotlon) 


POSTTEST(S) 
Date Score 


Comments/Areas 
of Difficulty 
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Appendix 6-B (Cont.) 
INDIVIDUAL PROGRESS INFORMATION - FORM TWO 



STUDENT: 




n1t Objective 
Brief Description) 
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Appendix 6-B (Cont.) 
INDIVIDUAL PROGRESS INFORMATION - FORM THREE 



STUDENT: 






Inlt Objective 
Brief Description) 
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Appendix 6-C 
G^OUf PROGRESS RECORDS 



As covered 1n Sectlo" 6 , there 1s also a variety of Informa- 
tion winch can be uf4l n tained on groups of students. Four 
forms which may be o f 1h this area are Included here. 

Forms One and Two <Jo nftt relate directly to tracking prog- 
gress, but do provide * means for collecting Information 
necessary to manage the activities of students, both on an 
Individual basis af<* *S a group. Form One 1s simply a 
method to Identify, fgr each goal, the related objectives 
and Instructional ma*eMals available for that objective. 
By using this form* * master 11st of all program goals, 
objectives and Instructional materials can be maintained 1n 
one location. Fori* can be used as a master 1.1st to 

Identify objective* Sfoss-referenced to activities and 
materials available t 0 provide Instruction for each objec- 
tive. 

Forms Three and Four * r e both examples of forms which can be 
used to maintain records over the progress of a group of 
students. In Form Three, each student 1s listed down the 
left-hand column. fOr e *ch objective there 1s a column to 
record the date of the student's pretest and score, as well 
as the posttest and *c0f6. in the abbreviated example below 
1t can be seen tha* h# Shaw has passed the pretest for 
Objective 1 and passed the posttest for Objectives 2 and 3. 
The second student, H % Leigh, has passed the posttest for 
Objective 1, but did n°t pass the pretest for Objective 2. 




Form Four provides a *ay to track progress simply by 
objective. Each stu<*e n t 1$ listed down the left-hand column 
of the form. For e*<;h objective there Is a space to 
Indicate the date th£ te*t was Passed and the score. In the 
abbreviated form b*l<>w 1t can be seen that B. Green has 
completed Objective* i, 2, 3, and 7. M. Hanney has com- 
pleted Objectives 1» and 5, while R. W1nn has completed 
Objectives 1 and 2. 
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Appendix 6-C 
GROUP PROGRESS RECORDS - FORN ONE 
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tfonat Materials Available: 
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tonal Materials Available: 
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Appendix 6-C 
GROUP PROGRESS RECORDS - FORM TMO 



JBJECTIVES: 
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Appendix 6-C 
GROUP PROGRESS RECORDS - FORM THREE 
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Appendix 6-C 
GROUP PROGRESS RECORDS - FORM FOUR 
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7. REQUIREMENTS AND RECOMMENDATIONS FOR EVALUATING 
CHAPTER 1 N OR D PROGRAMS 



INTRODUCTION 

This section of the handbook has been Included primarily as 
a place to Insert specific regulations and other information 
which 1s frequently referenced by the user. At a minimum, 
It Is suggested that a copy of the Chapter 1 Law and Federal 
Regulations related to N or D programs be Inserted 1n this 
section. Specific state policies, reporting forms, and 
other state-related documents should also be Inserted here. 

This section also presents a description of recommended 
Information which might be collected by each program for 
overall summary purposes. This description 1s followed by 
two examples of reporting forms a long, comprehensive 
version (Appendix 7-A) and a short version containing 
Information considered to be most essential (Appendix 7-B). 



RECOMMENDED INFORMATION TO BE COLLECTED BY PROJECTS 



Although there are no requirements for reporting evaluation 
results of N or D programs to the Federal government under 
Chapter 1 of The Education Consolidation and Improvement Act 
of 1981 (PL 97-35), there are some types of Information 
which might be collected by each program. This Information 
would be of use both to the program and to those who are 
Interested In finding out more about that program. For 
example, the project that collects the suggested Information 
will find 1t easier to accurately and completely describe 
the program to others. Similarly, should N or D project 
directors have an opportunity to share Information about 
their programs, the Information summarized in the following 
topics will provide a common base for Initiating discus- 
sions. 

1. Name of Inst1tut1on/Fac1 11 ty 

2. Address of Instl tut1 on/Fad 1 1 ty 

3. Name of Person Completing Form 

4. Above Person's Telephone Number 



Duration of Project 



A Chapter 1 project would generally be ex- 
pected to be 1 n operation for one year or 
less; the beginning and ending dates should be 
provided. 

Number of persons under 21 years who resided at the 
1nst1tut1on/fac1l1ty during the project, by age 1n years 
and last grade completed 

Age and last grade completed should be deter- 
mined as of the beginning of the Chapter *1 
project or as of assignment to the program, 1f 
later than the beginning of the project. The 
number should be an undupllcated count (I.e., 
each person 1s counted only once no matter how 
many times that person might be assigned to 
that 1nst1tut1on/fac111ty during the Chapter 1 
project. 

Number of persons under 21 years who partld pated 1n the 
project, by age 1n years and last grade completed 

Age and last grade completed should be deter- 
mined as of the beginning of the Chapter 1 
project or as of assignment to the project, 
1f later than the beginning of the project. 
The number should be an undupllcated count 
(see Hem 6) and, therefore, should be no lar- 
ger than the number of residents counted in 
Item 6. 

Number of project participants by radal/ethnlc group 

The suggested racial/ethnic groups are defined 
below. The total should equal the grand total 
in Item 7. 

• American Indian or Alaskan Native - a per- 
son having origins 1n any of the original 
peoples of North America, and who maintains 
cultural Identification through tribal af- 
filiation or community recognition. 

• Asian cr Pacific Islander - a person having 
origins in any of the original peoples of 
the Far East, Southeast Asia, the Indian 
subcontinent, or the Pacific Islands. This 
area Includes, for- example, -C-hlrva, India, 
Japan, Korea, the Phil11pp1ne Islands and 
Samoa • 
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• Black, not Hispanic - a person having ori- 
gins 1n any of the black racial groups of 
Africa . 

• Hispanic - a person of Mexican, Puerto R1- 
can, Cuban, Central or South American or 
other Spanish culture or origin, regardless 
of race. 

e White, not Hispanic - a person having ori- 
gins 1n any of the original peoples of Eu- 
rope, North Africa, or the Middle East. 

Number of project staff by job classification 

For each job classification, record the number 
of persons paid at least 1n part, by the Chap- 
ter 1 project 1n terms of full-time equivalent 
- (PTEs). To calculate the number of FTE staff 
members 1n a job classification, determine the 
number of hours each person 1n that job clas- 
sification worked per week. Add these numbers 
together and divide by the number of hours psr 
week that represents "full-time" for that job 
classification. If an Individual works 1n a 
Chapter 1 project for more hours than are paid 
by Chapter 1 funds, count the FTE 1n terms of 
hours worked rather than hours paid by Chap- 
ter 1. The job classifications are defined as 
f ol lows : 

• Administrative Staff - a person whose pri- 
mary assignment 1s to direct staff members 
or manage the Chapter 1 project and Its 
supporting services within an operating 
unit or facility (e.g., project directors, 
coordinators ) . 

§ Teachers - staff members who Instruct Chap- 
ter 1 students. 

• Teacher Aides - staff members who assist a 
teacher with routine activities associated 
with teaching and those activities requir- 
ing minor decisions regarding students, 
such as monitoring, coordinating exercises, 
operating equipment, and maintaining re- 
cords . 

• Othe-r - includes- curriculum specialists,- 
support service staff (e.g., social work, 
guidance, psychological counseling, health, 
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nutrition), clerical staff and others not 
Included above. Specify each "other" job 
classification and provide a separate 
count for each. 

10. Number of project and non-project staff by job classi- 
fication who received training funded by the Chapter 1 
project 

Non-project staff are those not paid at all 
by Chapter 1 funds. Job classifications are 
def1ne*d 1n Item 9. Numbers should be undu- 
p Heated counts (I.e., one person receiving 
training on more than one occasion 1s counted 
only once). Specify each "other" job class?- 
f 1 cat 1 on and provide a separate count for 
each. 

11. Number of project participants who received services 1n 
each project component area 

For each Chapter 1 project component area 1n 
which services were provided, specify the 
number of participants who received that ser- 
vice. The number should be a duplicated 
count (I.e., a. student shouH be counted once 
for each area 1n which services were re- 
ceived). Other Instructional areas Include 
English to limited English speaking students, 
vocational education, special education for 
handicapped, etc. Supportive Services In- 
clude social work, guidance, psychological 
counseling, health, nutrition, student trans- 
portation, etc. Specify each "other In- 
structional area" and each "supportive ser- 
vice" and provide separate counts for each. 

12. Number of project participants who received component 
services by duration of service 1n months and by last 
grade completed 

Last grade completed should be as of the be- 
ginning of the Chapter 1 project or as of 
assignment to the program, 1f later than the 
beginning of the project. Duration of ser- 
vice should be the total number of months 
each student received component services over 
the entire project. If a student's duration 
of service was an exact number of months, 
count that student in the first category con- 
taining that number of months. For example, 

1 f duration -of -servl c«- w-as -ex-act ly-~eigb.t 

months, count that student in the 7-8 cate- 
gory. The grand total should equal the com- 
ponent count 1n Item 11. 
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State the questlon(s) addressed for the purpose of 
evaluating the short-term effectiveness of the compo- 
nent. 

Refer to Sections 1 and 3 of this handbook 
for a discussion of purposes of evaluation 
and evaluation questions, respectively. Pro- 
vide only those questions which concern the 
short-term effectiveness of the component's 
services here. Other questions would be more 
appropriately discussed and addressed 1n a 
more comprehensive, final evaluation report. 

Check the 1nd1cator(s) of need for service used to se- 
lect the participants ~1n the component. 

Check one or more types of Information used 
to Indicate a student's need for the Chapter 
1 -project component's services. Test scores 
could Include achievement tests, diagnostic 
tests, affective measures, etc. Teacher judg- 
ments could Include ski 11 -def 1 dency check- 
lists, general referrals, estimated grade 
level, etc., and could be provided by non- 
Chapter 1 or Chapter 1 teachers. Other judg- 
ments could Include referrals made by other 
Instructional or support service staff, the 
students, etc. Other performance Indicators 
could Include grades, level of Instructional 
materials, previous participation 1n Chapter 
1 projects, etc. 

Check the type(s) of Instrument (s ) used to measure the 
Intended Impact of the component. 

Check each type of measurement Instrument 
used to assess the Intended Impact of the 
component services on the students. For ex- 
ample, a reading Instructional component 
would use some measure of reading achievement 
level; an affective component might use a 
self-concept scale; while a counseling compo- 
nent might employ Interviews or existing re- 
cords. 

On a separate sheet, provide the results of the anal- 
ysis of the effectiveness of the project component us- 
ing each of the Instruments checked above. Include at 
least the foTTowtngr — - 
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• a measure of the component participants 1 
average achievement level* performance, 
attitude, etc., after receiving compo- 
nent services (e.g., an average posttest 
score ) ; 

• a measure of the average achievement lev- 
el, performance, attitude, etc., expected 
of the participant had they not received 
component services (e.g., an average pre- 
test score); 

• the number of scores the above averages 
are based upon; 

• a brief description/identification of each 
Instrument; 

• the type of score used 1n the analysis 
(e.g., NCE, standard score, raw score, 
number of objectives, etc.); and 

• the date(s) when each Instrument was ad- 
ministered or completed. 



In the following appendices are the examples of the forms 
mentioned 1n the Introduction to this section. The first 
example (Appendix 7-A) 1s of a long, comprehensive form for 
collecting program Information. The second example (Appen- 
dix 7-B) 1s a short version for collecting Information con- 
sidered to be most essential. 
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Appendix 7-A 

SAMPLE FORM FOR COLLECTING DATA 
FOR CHAPTER 1 N OR D PROJECTS 

(Long Version) 

1. Name of Institution /Facility 

2. Address of Institution /Facility 



3. Name of Person Completing Form • 

4. Above Person's Telephone Number 

5. Duration of Project:* From To 

6. Number of persons under 21 years who resided at the 1nst1tut1on/fac1l1ty during 
the project by age 1n years and last grade completed at the beginning of the 
project or at commitment 1f later than the beginning of the project (undupH- 
cated count) 



[^^Jrade 
1 Age 


3 or 
under 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


10 or under 
























11 
























12 
























13 
























14 
























15 
























16 
























17 
























18 
























19 
























20 
























Total 
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7. Number of persons under 21 years who participated in the project by age 1n 
years and last grade completed at the beginning of the project or at commitment 
1f later than the beginning of the project (undupHcated count) 



^-^Grade 
Age*^-^ 


3 or 
under 


4 


5 


! 6 


7 


8 


9 


10 


11 


Total 


10 or under 






















11 






















12 






















13 






















14 






















15 






















10 























17 






















18 






















19 






















20 






















Total 























8. Number of project participants by racial /ethnic group 

American Indian or Alaskan Native 

Asian or Pacific Islander 

Black, not Hispanic 

Hispanic 

White, not Hispanic 

Total 



9. Number of project staff (paid at least 1n part by Chapter 1 funds) by job clas- 
sification (1n FTEs) 

Administration Staff . 

Teachers ~. — 

Teacher Aides "~. 

Other (Specify) ^ ~~ . 
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Number of project and non-project staff by job classification who received 
training funded by the Chapter 1 project 

Project Non-Project 

Administrative Staff 

Teachers ~~~~ 

Teacher Aides 

Other (Specify) 



Number of project participants who received services 1n each project component 
area (duplicated count) / 

Reading 

Language Arts 

Mathematics ~" 
Other instructional 
Ansas (Specify) 



Supportive Services 
(Specify) 
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COMPLETE THIS SECTION SEPARATELY FOR EACH PROJECT COMPONENT 



12. Number of project participants who received component services by duration of 
service In months and by last grade completed at the beginning of the project 
or at commitment 1f later than the beginning of the project 



Grade 
Duration 


3 or 
under 


4 


5 


6 


7 


8 


Q 


10 


11 

XX 


Total 


0-1 






















1-2 






















2-3 






















3-4 






















4-5 






• 
















5-6 






















6-7 






















7-8 






















8-9 






















9-10 






















10-11 






















11-12 






















Total 























13. State the question (s) addressed for the purpose of evaluating the short-term 
effectiveness of the component. 

Example: Do students who receive reading instructional ser- 
vices from the project make greater gains in reading 
achievement level than they would have made without 
that additional instruction? 



220 

7-10 



Check the 1nd1cator(s) of need for service used to select the participants 1n 
the component. 

Test Scores 

Teacher Judgments 

Other Judgments (e.g., self-referral) 

Other Performance Indicators (e.g., grades) 

Check the type(s) of 1nstrument(s) used to measure the Intended Impact of the 
component. 

Norm-Referenced Achievement Tests 

______ Criterion-Referenced Achievement Tests 

Other Tests, Scales and Inventories 

________ Observations 

_______ Questionnaires 

________ Interviews 

_______ Existing Records 

On a separate sheet, provide the results of the analysis of the effectiveness 
of the project component using each of the Instruments checked above. Include 
at least the following: 

t a measure of the component participants' average achievement lev- 
el, performance, etc., after receiving component services (e.g., 
an average posttest score); 

• a measure of the average achievement level, performance, atti- 
tude, etc., expected of the participants had they not received 
component services (e.g., an average pretest score); 

• the number of scores the above averages are based upon; 

t a brief description/Identification of each Instrument (e.g., In- 
strument name, edition, level); 

t the type 0/ score used 1n the analysis (e.n., NCE, standard 
score, raw score, number of objectives, etc.); and 

• the date(s) when each Instrument was administered or completed. 



221 

7-11 



Appendix 7-8 



SAMPLE FORM FOR COLLECTING DATA 
FOR CHAPTER 1 N OR D PROJECTS 



(Short Version) 



1. Name of Institution/Facility 

2. Address of Institution/Facility 

3. Name of Person Completing Form 

4. Above Person's Telephone Number 



5. Duration of Project: From To 

6. Number of persons under 21 years who resided 

. «it the 1nst1tut1on/fac1l1ty during the project 
(undupllcated count) 

7. Number of persons under 21 years who participated 
In the project (undupllcated count) 

8. Number of project staff (paid at least In part by Chapter 1 funds) by job clas- 
s1f1cat1on (1n FTEs) 

Adm1n1nstrat1ve Staff 

Teachers 

Teacher Aides 

Other (Specify) ~ 



9. Number of project and non-project staff by job classification who received 
training funded by the Chapter ? project 



Project Non-Project 



Administrative Staff 

Teachers 

Teacher Aides 

Other (Specify) 



10. Number of project participants who received services 1n each project component 
area (duplicated count) 

Reading 

Language Arts 

Mathematics 

Other Instructional — .... . 

Areas (Specify) 



Supportive Services (Specify) 
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COMPLETE THIS SECTION SEPARATELY FOR EACH PROJECT COMPONENT 



11. State the questlon(s) addressed for the purpose if evaluating the short-te rm 
effectiveness of the component. 



12. Check the 1nd1cator(s) of need for service used to select the participants 1n 
the component. 

Test Scores 

_ _ Teacher Judgments 

Other Judgments (e.g., self-referral) 

Other Performance Indicators (e.g., grades) 

13. Check the type(s) of Instrument (s) used to measure the Intended Impact of the 
component. 

■ Norm-Referenced Achievement Tests 

Criterion-Referenced Achievement Tests 

Other Tests, Scale*; and Inventories 

_ Observations 

Questionnaires 

_ Interviews 

Existing Records 

14. On a separate sheet, provide the results of the analysis of the effectiveness 
of the project component using each of the Instruments checked above. Include 
at least the following: 



o a measure of the component participants' average achievement 
level, performance, attitude, etc., after receiving component 
services (e.g., an average posttest score); 

• a measure of the average achievement level, performance, nt1- 
tude, etc., expected of the participants had they not received 
component services (e.g., an average pretest score); 

• the number of scores the above averages are based upon; 

§ a brief descr1pt1on/1dent1f 1cat1on of each Instrument (e.g., 
Instrument name, edition, level); 
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• the type of score used In the analysis (e.g., NCE, standard 
score, raw score, number of objectives, etc.); and 

t the date(s) when each Instrument was administered or completed. 
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8. RESOURCES FOR N OR 0 PROGRAMS 



INTRODUCTION 

When planning or conducting evaluation aclt1v1t1es 1t may be 
necessary to obtain some outside guidance. There are 
resources which can be extremely useful to project personnel 
who need some type of assistance. A primary resource for 
Chapter 1 programs 1s that of the Technical Assistance 
Centers (TACs ) 9 which provide consulting services, at no 
direct charge 1n the area of evaluation* The major focus of 
this section 1s to describe the types of services available 
through the regional TACs. An additional resource which 1s 
briefly discussed 1s the National Diffusion Network, which 
disseminates Information about successful educational 
programs. 



TECHNICAL ASSISTANCE CENTER SERVICES 

The United States Department of Education has established 
regional Technical Assistance Centers (TACs) throughout the 
country to provide consulting services at no direct charge 
to projects funded by ECIA, Chapter 1. Depending upon the 
needs of those being served, TAC services may take a variety 
of forms r Including: (1) on-site visits to deal directly 
with evaluation problems affecting the project; (2) local 
workshops dealing with specific Issues; (3) telephone 
consultations to answer specific evaluation questions; and 
(4) pr'lnt and mediated materials relevant to Chapter 1 
evaluation 1 ssues • 

Personnel from regional TACs are available to provide help 
1n a variety of topics relevant to Chapter 1 evaluation 
Issues. Some areas of particular Interest to N or D 
projects might Include: 



o Developing procedures to meet evaluation and 
reporting requl rements ; 

• Selecting students for a program; 

§ Developing long-range planning procedures; 

• Selecting tests and other Instruments; 

• Interpreting and using test results; 
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f Using appropriate data analysis procedures; and 

e Applying microcomputers for evaluation and man- 
agement. 

Frequently requested topics for workshops include: 

e Selecting Criterion- and Norm-Referer ced Tests 
o Systematic Teacher Ratings for Needs Assessment 

• Developing Composite Scores for Student Selec- 
tion 

e Reporting Evaluation Results 

t Conducting Descriptive Evaluations 

§ Evaluation for Program Improvement 

• Developing Tests 

e Selecting Measures of Affective Behavior 
e Functional Level Testing 

• Developing Objectives for Program Evaluation 

• Strategies for Program Improvement . 

• Describing Program Characteristics 
e T1me-on-Task 

e Quality Control 

• Sustained Effects 

• Test Administration and Scoring 

• The Joint Dissemination Review Panel Process 

• Evaluating N or D Chapter 1 Programs 

For more detailed Information on workshops and other 
consultation services the regional TAC should be consulted. 
The TAC regions and contractors are Ustad in Appendix 8-A. 
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There are some services which are not available through the 
TACs. These Include the enforcement of Chapter 1 rules and 
regulations, the promotion of particular standardized tests, 
the gathering of data for studies, the writing of state or 
local evaluation reports, the recommendation of curriculum 
or program revisions, and the analysis of evaluation data. 



TAC Clearinghouse Resources 

The Technical Assistance Center Clearinghouse 1s located 1n 
the Region IV TAC, at the Northwest Regional Educational 
Laboratory, Portland, Oregon. The Clearinghouse 1s respon- 
sible for collecting, cataloguing and distributing research 
documents and workshop materials relevant to Chapter 1 
evaluation. It should be noted that materials 1n the 
Clearinghouse range from subjects of general Interest 1n 
regard to Chapter 1 evaluation to very specific topics. 
There are also some Items which directly address N or D 
programs and Issues.. To obtain more information about the 
Clearinghouse materials, contact the regional TAC office. 



Available Materials 

Each regional TAC can also provide copies of various 
documents relevant to Chapter 1 evaluation, Including the 
evaluation chapters from The Pol 1 cy Manua 1 , The User's 
Guide, and The Eva lua tor's References" ! The last item i s a 
col lection of technical papers, which includes the following 
titles: (l) An Overview of Hazards to Avoid 1n the Title 
I Evaluation and Reporting System; (2) Selecting Students 
for Title I Evaluation Projects; (3) Composite Scores; (4) 
Selecting a Norm-Referenced Test; (5) Local Norms; (6) Using 
Non-Normed Tests 1n the Title I Evaluation and Reporting 
System; (7) Out-of-Level Testing; (8) Test Floor and Ceiling 
Effects; (g) Collecting Achievement Test Data; (10) Factors 
That Influence Test Results; (11) Problems with Grade 
Equivalent Scores; (12) Interpreting NCEs; and (13) Score 
Conversions . 

Each regional TAC also maintains audiovisual materials and 
print packages which, upon request, are available for loan 
to state, local and project personnel, as well as to other 
groups. For more specific Information regarding print or 
audiovisual materials, contact the regional TAC office. 
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THE NATIONAL DIFFUSION NETWORK 



The National Diffusion Network ( NDN ) , sponsored by The 
Education Department, 1s a program to Identify and dissemi- 
nate successful educational practices. NDN publishes an 
annual document, "Educational Programs That Work," which 
describes promising programs. These descriptions, by 
project, Include: (1) characteristics of the target 
audience; (2) a brief explanation of the program processes 
and materials; (3) Information on evidence of effectiveness; 
(4) requirements for Implementing the program; (5) financial 
requirements for Installing the program; (6) services 
available from the project to those Interested 1n adopting 
the program; and (7) a contact person for further Informa- 
tion. 

Obtaining descriptions about exemplary programs can be 
useful for both evaluation activities and program planning. 
Exemplary programs do need to document their effectiveness 
through some type of sound evaluation procedures. Obviously 
then, these programs could be a source of Information on how 
to plan and conduct evaluations. Likewise, if student 
selection procedures are an Issue, a review of how a similar 
program selects students for Its program could be a source 
for new approaches. In the area of program planning, 1f the 
evaluation results Indicate that Improvements need to be 
made In the curriculum, then programs described by NDN may 
be a souree for r.ew Instructional approaches. 

Each state has a person who serves as the NDN State Facili- 
tator. These Individuals are a good source for the Identi- 
fication of programs which may be of Interest and for more 
Information on conferences, newsletters and other publica- 
tions which focus on exemplary programs. The State Facili- 
tator 1s also the person to contact for procedures to follow 
or help 1n adopting an NDN program. 
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Appendix 8-A 

TECHNICAL ASSISTANCE CENTER REGIONS AND CONTRACTORS 



Region I Includes: Maine, Vermont, New Hampshire, Massachu- 
setts, Connecticut, Rhode Island, New York, New Jersey, 
Puerto R1co, Virginia, Delaware, Maryland, the District of 
Columbia, Pennsylvania, West Virginia, and the Virgin 
Islands. 



The prime contractor 1s 

RMC Research 
400 Lafayette Road 
Hampton, NH 03842 
(603) 926-8888 



The subcontractor 1s: 

Educational Testing Service 
Princeton, NJ 08540 
(609) 734-5117 



Region II Includes: Iowa, Illinois, Indiana, Missouri, 
Wisconsin, Kansas, Ohio, Nebraska, Kentucky, North Dakota, 
South Dakota, Michigan, and Minnesota. 



The prime contractor Is: 

Advanced Technology, Inc 
1 Park Fletcher Building 
2601 Fortune Circle East 
Indianapolis, IN 46241 
(317) 244-8160 



The subcontractor 1s: 

RMC Research 
9300 West 110th Street 
Overland Park, KA 66210 
(913) 341-0008 



Region III Includes: North Carolina, South Carolina, 
Georgia, Florida, Mississippi, Alabama, Tennessee, Louisi- 
ana, Arkansas, Oklahoma, and Texas. 



The prime contractor Is: 

Educational Testing Service 
250 Piedmont Avenue, NE 
Suite 2020 
Atlanta, GA 30308 
(404) 524-4501 



The subcontractor Is: 

Powell Associates, Inc. 
3724 Jefferson Street 
Suite 205 
Austin, TX 78731 
(512) 453-7288 



Region IV Includes: Alaska, Arizona, California, Colorado, 
Hawaii, Idaho, Montana, New Mexlct, Nevada, Oregon, Utah, 
Washington, and Wyoming. 



The prime contractor 1s: 

Northwest Regional Educational 

Laboratory 

300 SW 6th Avenue 
Portland, OR 97204 
(503) 295-0214 
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